As anyone who has used Linux systems for production systems knows all too well, there's an art to arriving at a stable configuration with all dependencies met. Linux distributors do an excellent job of delivering systems that meet this criteria, and keeping them there through their update processes as functionality updates, bug fixes, and security updates get laid on top of the out-of-the-box system. The amount of work and the success that they have in delivering both the base distribution and the stream of updates that follow is widely unappreciated.

When it is appreciated it's usually because the standard distribution and kernel configuration isn't suitable for many commercial applications, and one or more custom configurations has to be developed and maintained by the organization's own Linux support group.

The reasons for having to develop custom configurations are as varied as the organizations themselves, but mostly it boils down to a handful of reasons. First, the generic distributions contain support for all of the ancillary services that come with the distribution, for all of the free applications and utilities that come on the stack of CDs accompanying the basic Linux system installation, and in a configuration that will run on the wide range of hardware systems and peripherals that any random buyer of the system may want to load it onto. Some careful "pruning" of the applications, services, and support infrastructure can lead not only to a smaller and faster system but to a more reliable and secure one too.

The second common reason is that in building their standard distribution distributors make decisions about the way things are built or configured, and this then affects other parts of the system relying upon those items. If the decisions made by the distributor don't match the requirements of the organization deploying the system, making the appropriate changes can have a cascading effect throughout the system. Those changes also mean that the standard upgrade patches supplied by the distributor probably can't be blindly applied.

The third principal reason is that services or applications not included in the standard distribution may have to be accommodated. These may make specific demands on the kernel requiring some non-standard tuning. System-level support in the form of various Open Source or third-party libraries and related files may also have to support those non-standard services or applications. These changes can lead to various dependency resolution problems that will have to be solved, so it's often more complex to build a system extended this way than it might appear at first.

Whatever the reason and whatever the changes made, various decisions will be made in the construction of the final system. This is where danger lurks.

Open Source software and many of the tools it uses have a property that's a two-edged sword: there's almost always more than one way of doing something. This can be incredibly valuable, since one way may be a much better match for the application's requirements than any of the others. More mundanely, the choice might be made by the individual or team customizing the distribution simply because they're more familiar with that way of doing things or find it easier.

Problems begin to happen when you have multiple support teams, each building the distribution to be used in their data center in their own way. This is not unusual in larger organizations, particularly those with geographically distributed data centers, and even more particularly if those data centers are in different countries, there may be more than one Linux support group.

I recently ran across an example of this when consulting for a large multinational company on a problem it was experiencing with strange behavior from a third-party application it was running. The company uses Red Hat servers and has a "standard configuration" of its own design that's used to build the servers in each of its data centers (US, Europe, and Asia). The problem turned out to be a plug-in in the commercial application. The plug-in was compiled in the US and binaries given to each of the data centers to include in the application. This was fine, except that each data center built their distributions themselves, and although they ended up with kernel configurations looking very similar, and package lists that looked similar, there were enough subtle differences to make the plug-in binary behave differently.

Although it may have been possible to take the purist Open Source approach and distribute the source to each data center and let them build it on their configuration, which would probably have fixed the problem, the company decided that since this was part of a business-critical infrastructure that just had to run non-stop, this was one risk that it could avoid by implementing good configuration management and having a single build for systems that were always supposed to be identical. This would not only solve the problem, but head-off any similar problems that might crop up.

Configuration management, along with real QA, is an area that is often under-funded and doesn't get the attention it deserves. This extends across companies from the start-up phase, to multinational mega-companies. It's a false economy. Configuration management isn't that hard to implement, and there are several excellent toolsets on the market that can not only help keep control of the several different configuration that most organizations find themselves running, but also help with deploying and upgrading those systems.

The cost of dealing with problems ultimately traceable to configuration differences, which not only includes the time taken to track down and fix the problem but also the affects that any downtime of critical applications can end up making the costs of implementing and maintaining a good configuration management system look like a really good investment.