Control Freak

Whilst there are plenty of people who mistakenly follow a technical religion, there are actually some very good technical reasons for minimising your use of Microsoft Windows. The main one of these is the horror of the Windows Registry.

At the heart of Windows lies a group of binary files that store almost all the configuration information for both the operation sytem itself and the installed applications, collectively called the Windows registry. Other operating systems have individual configuration files, sometimes binary and sometimes text.

The registry has lots of problems:

• It can’t be edited by hand with a text editor, a specialist tool is needed. I realise this applies to other configuration systems (like OSX plist) and I would make the same comment about them.
• Spurious entries and corruption in the registry can cause slowness and operational problems. It’s for this reason that a great market has grown up in registry cleaning software.
• Corruption is non-obvious to spot. This is caused partly by the binary format and partly by the cross-linking between various parts of the registry. This is another reason why registry cleaning software is needed.
• The configuration information for a particular application appears all over the place, not just one place in the registry, so you can’t easily see all the information you need at once.
• It is not easy to just swap between two configuration sets for an application without considerable hassle. With config files it is of course easy enough to just keep various copies and move them around as needed.
• De-installation of applications is a nightmare so normally only the installing application can do it since the bits can be all over the place. Of course if you don’t have the installing application, or it does not have a good de-installer then we need a registry cleaner, again.
• The registry is a single point of failure. If you lose one of these files (say the software one) then you lose all the configuration information for all applications.

Therefore, the impact of the registry compared to separate config files is:

• It takes a lot more time to get anything done.
• It takes much more effort to learn.
• The routes open to a sysadmin to test and debug are much more limited compared to other OSs.
• When it goes wrong, it take much longer to fix.
• It ends up costing a lot more money to support.
• Systems are less stable.

All very good reasons for just sticking with separate configuration files, as other operatings systems do, preferrably in text format.

This is one of the most difficult problems to deal with – when people get religion – in other words when people get extremely strong views on technology because that’s what the entrails of the chicken told them.

Now don’t get me wrong, I’m not equating strong views with a belief in voodoo, it is actually much more complicated than that.  There are plenty of people with strong views who have detailed reasoning for those views.  The issue is those views that are more religion than science.

For example, there are plenty of Windows avoiders out there.  But there is a real difference between those that avoid Windows because they see them as the evil empire and those that avoid them because they understand just what a disaster the Windows registry is.

So whenever I encounter any strong views I have to dig down to see if there is reasoning behind it to separate it from religion.  On the surface the distinction is not obvious.  Occasionally I find that the strong views are based on an intuitive understanding of the issues that can’t easily be put into words.  In these cases the process of digging down can sometimes lead to a switch from religion to science (a deconversion?).

The reason this bothers me so much is because I generally find that those for whom this is religion, can unpredictably change their views and worst of all, I don’t think they really understand technology.

At the same time I’m slightly concerned when I meet people who have been working in IT for several years and yet don’t have strong views.  We all come across so much rubbish that anyone who fails to be polarised by it is probably not paying attention.

For some years now I’ve viewed the practice of users storing data files in directories on local hard disks or even servers as completely anachronistic. In fact I think of it much the same way as I think of data held on a floppy disk – I know it’s there but getting at it is so hard it might as well not be.

To me, all this data is isolated in islands of information and needs to be rescued by being stored in a different way.

What I want is for all information to be:

• Searchable
• Categorised, with multiple categorisations
• Catalogued. In other words described in a table of contents
• Versioned
• Accessible by anywhere on the net in a controlled way

and that is not done by current operating systems. So, for that reason I think we need to see a shift towards all data being stored in specialist applications that provide these functions. Yes, I realise that desktop search has dealt with the searching issues, but it still doesn’t tackle my other points.

The product I prefer for this at the moment is Lotus Notes, but the moment something better comes along, I’ll switch. For example if there was an Internet based app that combined webdav, caldav, email, opendoc format files and so on then that might do it.

The final thing to note is that in any organistion the highly structured information is already stored in a specialist application, called a database. All I’m talking about is taking the same approach to loosely structured information, which is actually a first step in a knowledge managememt strategy.

I talked in an earlier post about people’s misconception of the management big stick, i.e. the ability to just instruct someone to do something rather than come to an agreement with them that they will do it. The misconception that I was highlighting was the belief of some people that wielding this big stick is what management is all about.

The truth is that the more I do this the more resented and thereby ineffectual I become. Only a little bit of respect is lost each time, but it soon adds up.

There are times when it can have the oppposite effect, but those are largely when I’m using the management override (to give it a more polite name) to impose a view that is almost a consensus, just one or two intransigent people are holding out.

So here are my tips on how to wield the management override:

Only ever do it when the situation absolutely requires it.

I have come across far too many managers who invoke the override at every point where they believe that people need directing to do something better than they currently are. In other words they think they know best and so they make people do it their way.

In many cases the manager will actually be right, their greater experience and awareness of the situation giving them the edge. But in these situations I ask myself the question – do I actually need to use the override or is there a better way to get what I want? In almost every case the answer is that I don’t need the override. Instead I can explain to the person concerned why I think there is a better solution and let them make their own mind up. Doing it this way means I have treated them with respect, like a fellow human being, not like a machine.

Another question I ask myself is – is this sufficiently important for me to want it changed? Surprisingly the answer I keep coming across is no, it isn’t. Again, there are some managers who have to fix everything they think is broken, no matter how important, who is doing it, or what the possible outcome, and they fix it by using the management override. I think that what they forget is that we are all learning and we all learn by making mistakes and getting it right the next time. If I stop someone from ever making a mistake then they might end up learning what I teach them, but they will never learn to learn for themselves.

This is one of the biggest generators of resentment for management – if we don’t let people get on and do their jobs because we interfere too much.

If it has to be done then it should be done as respectfully as possible

There are times when we have to do it. For example I do it about the use of indirection in our infrastructure because I think that is so overwhelmingly important. But when I do it there is a certain process that I follow to minimise the impact.

• At the forefront of my mind is the knowledge that I am overriding the will of others, and their views are based on just as much thought as mine and just as deeply held.
• I explain my reason for doing it, which is normally because there is no consensus and yet a decision must be made, or because I’m convinced this is the right way to go.
• I explain the decision in as much detail as possible to those affected.
• I generally need to publicly state that my solution is not perfect and there are unresolved issues. Then I need to make it clear that I know those issues and will keep a watch on them.
• I have to make it clear that if a better solution comes along, in particular one that has consensus, then I will change my views. The door is not closed.
• But sometimes I have to make it clear that the fact I am using the override does not mean they can disengage and leave things to flounder. Of course good staff would never do that.

Other tips

When I’m involved in a discussion about what is the best way forward, or how something should be done, I have to make it clear that I am participating on an equal basis as others and not using the override. I have to repeat this often. If I don’t then people forget and assume I am using it just because of my position and not anything I say.

Occasionally, despite my being convinced I am right, I find that I have got it completely wrong and so if I had used the override then I would have caused a real problem. I wonder how many managers do that and then don’t have the integrity to admit it and try to undo the mess they’ve created?

When I used to be a programmer I was very hot on my own internal naming conventions and I was using my own variant of Hungarian notation long before I discovered there was such a thing. Writing systems by myself meant that I could enforce this internal consistency and make my life simpler. But I always longed for my conventions to be understood by the compiler. So, just by giving something a specific name the compiler would understand how to connect it to other code, without me needing to explicitly configure it.

I’ve now seen that in action in Ruby on Rails and I am well impressed. It even has a name – ‘convention over configuration’.

An example of how it works in Rails is simple enough. If you have a URL http://domain/x/y then Rails expects to invoke a method called y in a controller called x. So easy. But Rails does this in a few more places and is always looking to increase use further.

I find that this approach makes so many things easier that it should be considered as a general principle for software design, not just a quirk of Rails. The benefits that it brings are:

• You have to write much less code.
• Spotting a naming mistake is simpler because it stands out more.
• If you don’t know the name of a particular function then you have a lot more chance of making a successful guess.
• You have more chance of guessing how and where a function is used just from the name and context.

Brains are naturally pattern matching machines and I think convention over configuration appeals to them much more than a long XML file. I’m surprised I don’t see it being used much more often but I bet we will in the future.

A small bugbear of mine. Why do auditors ask me every year “Do you force your users to change their passwords regularly?”. Since when did this become such an unchallenged tenet of security?

Well I don’t agree that this is a good idea at all, and I patiently explain that to the auditors each year. In fact I take the view that forcing people to change their password regularly can actually reduce the security of most systems. My reasons for this are fairly clear:

• Most people find passwords difficult to remember and the more complex the password, the more likely they are to forget it. This is probably because there is no obvious handle for the memory to be hooked onto, i.e. no mnemonic, no event, no place etc.
• If you force people to change regularly then you also force them to develop mechanisms for storing their passwords other than just remembering them, because the burden on their memories is just too much.
• These mechanisms are nearly always insecure. In some cases it is a post-it note stuck to the screen, sometimes a note in a drawer or on the desk or even a text file on their computer.
• Choosing a password is actually quite hard for many people. They can’t think of anything and so they tend to go for simple, memorable names. Just the kind of thing that is vulnerable to a dictionary attack
• Once someone has finally remembered a password, generally only through repeated use, do they then destroy the physical record of it.

So what really matters, is to teach users

1. How to choose a strong password. I’ve seen some nice online tools that evaluate the password strength as you type it. I’m sure they’re a bit corny and generally don’t include dictionary searching, but the do help the user.
2. How to choose a memorable password, so they don’t have to record it anywhere.

Now if auditors asked me whether we do that or not, then that would be a sensible question to ask.

Submarines are built with compartments separated by bulkheads. These prevent the spread of fire and limit the impact of damage to one compartment. Good technical architecture follows exactly the same principle, but it normally applies to:

• Compromise of one system leading to compromise of other(s)
• Failure of one system leading to failure of other(s)

Interestingly, wherever this principle is applied there is a tradeoff, normally in managing it, and this limits just how far to go with it. From experience I don’t think people go far enough.

Networks

The most obvious application of this is with networks where it is quite common to split them into VLANs or even separate LANs with firewalls or routers acting as the bulkheads. Access between them is limited and specific thought is given to what happens if one is compromised.

The tradeoff is that this requires more kit, more management and things that would otherwise just work have to be allowed to work. But these are generally such small tradeoffs that no professional omits firewalls for those reasons.

Servers and Applications

Less common, outside the world of financial insitutions and other high value targets, is the separation of applications between different servers. I joined one organisaton where this principle had been applied fairly stringently for some years to the position of only allowing one or two application ports open per server. Whilst extreme, I think it was a major contributor to the lack of downtime and systems failure.

We all know that different applications on the same system can interfere with each other, particularly in a Windows environment with shared libraries and poor memory protection. But tracing such problems generally requires special tools and a thorough investigation. If a fault only occurs very occasionally then this analysis is rarely done. From working in an environment with applications so completely partitioned on different servers, I think the impact is dramatically reduced failure rates.

Of course there are sound security reasons for this separation as well. If one application is compromised then it is much more difficult for that to lead to compromise of other applications if there aren’t any.

The major tradeoff here is that many servers are left grossly underused and that has knock-on implications, particuarly for space, power and air-con in the datacentre. These are increasingly important concerns and I would urge anyone to resist allowing those concerns to dictate server planning. Just buy more racks, supply more power and fit more air-con.

Passwords

Even less common is the separation of passwords (or keys) into domains. This is only about security and it is the application of submarine compartments just as with networks. If someone manages to crack a password then where else can they use it? The same thing goes with keys, specially ssh keys.

Passwords, and keys in particular, are much more carefully guarded than most information because the inherent security value of them is instantly recognisable. Despite that, I doubt that many people do a formal analysis on what happens if a password or key is compromised – just how far can people get and what is there to stop them?

Conclusion

That leads me on to the final point that this principle is not one that needs a blanket imposition, but rather a risk-based analysis to determine where best to use it. This is undertaken by asking the basic questions of ‘what else can be affected if this system fails?’ or ‘what other systems can be targetted if this system is compromised?’. Asking these questions can produce some uncomfortable answers.

Delegation and empowerment are two very trendy buzzwords used by modern managers. But I’m not sure that many people, both managers and staff, have anything more than a vague notion of what they mean in practice.

Every now and then someone comes to me and says,

“I’ve got this really good idea how to sort out this problem. It means us changing the way we work like this …”

to which I reply,

“Great idea. As you’ve thought of it I’d like to delegate managing this change to you.”

Now a very small percentage of people can’t wait to go and tell everyone else what to do, but most people are horrified at the idea and instead reply with either

“Thanks. So can I go and tell everyone they you’ve decided they have to change and if they have any problems to talk to you?

or

Thanks. So will you go and tell everyone that you’ve decided they have to change?

All three of them are making the same mistake, which is to assume that a management decision is about telling people what to do. So they react according to their personality when they think this is what they are being asked to do. I’ve learnt to spot a flicker in the eyes that gives away this internal conflict (or in some cases the eyes lighting up at the prospect) and explain exactly what I expect.

The steps are:

• Start off by working out who will be affected by the change. Not just colleagues but managers as well.
• Then consider how they are likely to react to the change.
• If it looks like they will have some issues, then work out what to do about it
• Then go and consult with the people who will be affected, explaining the change, listening to their views armed with the planning you have already done.
• If there are any concerns, suggestions or other views, then listen to them and change your idea to accommodate these views. Don’t be stubborn, negotiate.
• Aim for consensus, and when you have it then the get a firm plan agreed for implementation.
• Keep the people affected informed every step of the way

And that’s all there is to delegation, empowerment and decision making – reaching consensus amongst colleagues and keeping people informed.

The most important thing I want them to realise, is that none of this takes the official role of ‘management’ to achieve. You can do it whether you are the most junior person in the team or the most senior, all it takes is good listening, persistence and a desire to get things done. Interestingly, it was following these steps before I was a manager that got me recognised as someone who got things done and helped me get promoted to management.

Unfortunately there are quite a few managers who don’t realise that this is how they should reach decisions, by consensus, not by the management big stick. I had one manager working for me a few years ago who was so bad at this that I had to draw a diagram in Visio to teach him how to make decisions without trampling over all his staff. He never really got it and was both unpopular and ineffectual as a result.

It is increasingly common to find that applications have remote API access. With the ubiquitous use of web technology people can add XML based remote interfaces quite quickly.

The temptation is now there to connect together all sorts of systems using these APIs in an event driven fashion. For example, if we want to connect our customer database to our mailing list system, which has a web based API for managing users, then this can be easy to set up. Whenever there is a change to our customer database this triggers some code that uses the mailing list API and we have the change propagated in real time. Seeing systems connected together like this with real time interaction is addictive.

If we’re not careful though, we end up building tightly coupled systems when this might not be what we want.

To explain this, let’s talk about a sending system and a receiving system and go through some scenarios:

What happens if we want to take the receiving system down for an upgrade?

Hopefully that’s something that was planned for and the sending system can detect if the receiving system is not available and queue the changes. Then when the receiving system comes back on line the sending system can process the queue. So that one was easy enough.

What happens if the receiving system dies and has to be restored from a backup tape?

Now that’s a bit more complicated because the changes might have been lost. To get around this all we need do is ensure that every change is written out to a table or file of changes. That way we can replay the changes lot by restoring from backup to get the receiving system back to the right state.

What happens when the format of the changes, changes (if you see what I mean)?

This happens all the time. The format changes so we change the code that fires off the changes but forget to change the code that stores the changes. Either that or we do remember but make a mistake in the storing of the changes. Well this is also easy to get around, we just ensure that the process on the sending system that sends changes to the receiving system uses as it’s source information, the stored changes after they are stored. That way, there is only one source and we can’t introduce an error we don’t find until much later.

What happens if for some reason we don’t want to send the changes?

Let’s just imagine the receiving system develops a fault in the API, say after an upgrade, and trying to use the API causes it to crash. Obviously we don’t want to turn off the sending system so we could firewall the receiving system, but that’s a bit of a pain and may not be possible. The obvious thing to do is to turn off the process on the sending system that fires off the changes.

And there you have it, a loosely coupled system. All it takes is for us develop the sending system so that it:

• writes out every change to a table or file of changes
• uses the written out changes as the source of data for publishing those changes
• makes the process that publishes those processes a separate process that can be started and stopped independently of any other systems
• enables that process to start from any point in the stored list of changes so that you can replay changes.

The final thing to point out is that this has an unexpected benefit. If we want to test a new version of the receiving system API then we can just set the sending system process to replay the last few days of changes and see if the receiving system copes. No special code needed.

One of the most unusual policies that I operate in my technical team is to let them buy whatever books they want provided they are at least loosely work related. All they have to do is send an email to purchasing and they will have the book delivered, no questions asked.

I don’t mind if they might start reading the book, decide it is rubbish and stop reading. Nor do I mind if they want a book that their neighbour has on their desk. People read books in different ways.

The rationale is

• the more they know the better they will be at their job
• in order to do their job they have to keep up to date with new things
• books are a very cheap way of learning
• having a book nearby as a reference can be an enormous timesaver

Most people on this scheme spend around £250 per year, but even the most determined person won’t exceed £1,000 per year on books, which is about the same cost as one third of an average technical training course. For a 25 person team I budget around £5,000.

You might be wondering how do you stop technical staff from building a library at home? The answer is that I don’t actually care if they do. Technical books are not, on the whole, like classic novels. Nobody reads a FoxPro manual any more because it is stale information. They are much more transient, almost disposable.

But we do stamp all the books with a ‘property of …’ stamp, just to limit their resale (not that anyone in my team would do that) and to remind people whose books they are.

Interestingly, with this policy in place, I find myself getting suspicious of those in the team that don’t buy any books. Can you really keep up with new technology just by reading the web?