Linux/Unix application configuration methodology

We are experiencing substantial application sprawl at my current employer while staffing either stays stable or decreases.  The combination of 9-5 normal development/operations/support combined with more after hours time for production operations is leading to increased human error and general difficulty in managing the movement of application & settings between dev/test/prod environments.  A contributing factor is the fact that each application has a slightly different method of handling configuration data.

This lack of a standard template and methodology for handling configuration was not sustainable.  This lead me to identify a number of core issues and work toward a process the mitigates them.  Some common problems include mismatches of configuration data between servers in a specific environment, changes not getting propagated between environments and incorrect configuration propagating between environments.

The mitigation methods include:

• Use of a source control system (Subversion) to contain all application configuration/settings data.
• Configuration files named based on server name/environment - typically for properties files
• Spring XML configuration files keyed by environment
•  Environment setup broken out into up to 3 distinct files - one for each of application, server and environment.  These files are named based on the type of data.  The general idea here is that application wide settings reside in a shared application wide (all server, all environments) file.  Any environment specific profile setup is in a file named based on the environment type.  Each server has a file based on the server name and is primarily responsible for setting the environment type.
• A shared profile processing driver that each server uses to "source" the appropriate configuration files.  There are some utility functions shared between all systems. The application wide environment setup profile plays the role of the driver and used the utility functions to process the server specific environment profile first, followed by the environment specific profile and finally completes its processing of the application wide environment settings.
• Utility functions include an item which displays the server name, application name and environment type.  This is called at login and is available to operators to quickly verify they are operating on the system they expect to be on.

This methodology is combined with the use of NFS in production and sometimes non-production environments.  There is an additional benefit of this combination in the production environment.  There is no direct network server to server communication between the production and disaster recovery (DR) environments but the production NFS storage is replicated to the DR site.  By including all DR environment settings/configuration on the production NFS storage, it automatically gets replicated out to warm DR servers.  Some additional shell scripting, a couple cron jobs and a standardized but special app deployment method result in the ability to have the DR application/web servers automatically deploy the production application versions in an unattended fashion.

I am still evaluating this mechanism in combination with Jenkins jobs and remote nodes for a more automated production deployment. 

Overall, there is a little learning curve for my team but this is already providing some benefits including traceability of change, identification of unapproved changes (run Jenkins jobs which check for correct versions of settings and local server modifications) and easier difference identification/handling via Subversion diff/merge functionality.

One other item of note is that the Subversion "externals" functionality is somewhat key in the sharing of some items which need to be available to all applications but would be awkward to manually synchronize or duplicate.