Application Server Deployment

What are drivers of an organizations deployment strategy? 
 The ones that come to mind are

1. Specific technology investments such as WebLogic (think of cluster deployments).
2. Generic process based on common technology (think of Tomcat Manager)
3. Custom processes, infrastructure and configuration to meet specific needs

Which is appropriate? I think the answer is "It depends".

If an organization has an investment in WebLogic or other costly technology and the deployment capabilities meet it's needs then maybe there isn't a need to do anything different than the product dictated solution. Using WebLogic or similar solution likely means there are multiple tiers and maybe load balancing of one or more tiers over multiple servers.  That type of environment is substantially more complex to setup and maintain than a single tier/server solution.  If the organization is staffed appropriately and has the proper training with the specific tools then doing something different seems unwise unless there is a good reason.  The main downside to this solution is cost - for both the tool and either training or hiring of the appropriate staff.  If the application hosted is critical then the costs multiply as well since you likely would want multiple staff trained to provide coverage during vacations or other unplanned emergencies.

Many small/mid-size organizations likely use Apache Tomcat or something similar.  There are reasonably well documented install and deployment procedures available on-line or in various books.  For organizations with smaller or less trained IT departments this is certainly a workable solution.  Most developers are more than capable of getting a default install up and running and managing it.  Applications can be scaled by load balancing multiple servers which starts to increase the complexity but is likely manageable at smaller scales.  After a certain point, scaling via adding more load balanced servers and using the default deployment procedures starts to become fragile.  Examples of the result of the fragility might include failed deployments, individual servers return wrong results, servers accessing wrong resources or what can be best described as flaky behavior.  These problems likely stem from things such as improper change management, rushed planning, lack of reliable deployment tooling/procedures, tight maintenance windows, etc.

Are there alternatives to the above and why would an organization use them?  Yes there are alternatives to the above scenarios and I will document one here.  This is only an alternative and is not necessarily the best one for every organization.  Each and every organization has to evaluate their resources, risks and requirements to determine what is appropriate for themselves.  This alternative focuses on reducing deployment errors, speeding up the deployment process for some cases and some other benefits I will document.

The assumptions for this alternative are:

1. mid-size organization
2. understaffed IT with wide duties
3. critical applications with at least a moderate rate of change
4. basic web application (multiple load balanced web servers and 1 DB server)

The basic technology used for this particular solution are:

• NAS NFS storage
• Apache Tomcat
• Linux based Web servers

Since many deployment errors can be traced back to mistakes updating configuration data manually across multiple servers.  The basic solution is to externalize the primary configuration data to a location outside of the web application where it resides on NFS storage.  Primary configuration data means data which differs between the production and non-production environments - such as DB connection/pooling info, special URL's, etc.  By moving the data outside of the application and web server and storing in a single location you reduce the touch points during future deployments.  If the primary configuration doesn't need to change then you don't need to edit those files during normal deployments.  This reduces the risk of unintended changes.  This can also speed up deployments since you may only require a restart of the remaining web server processes once the shared configuration change is made.

Another improvement to the overall environment includes installing the Java JDK on NFS storage and having a Linux soft-link (in a location on the NFS storage) to the current JDK in use.  Any later JDK upgrade process is faster because you only need to do one install, change one soft-link and start the application up.  These last 2 steps are the only steps where the application should probably be down.  If you have a lot of servers to deploy to, this can be a real time saver.  This also saves some space but storage is relatively cheap so the benefit is likely minimal.

Another idea is to install the Apache Tomcat software on NFS using the delivered ability to specify CATALINA_BASE and CATALINA_HOME separately.  After doing this, setup a Linux soft-link (in a location on the NFS storage) to the current Tomcat.  Each web server would reference the soft-link instead of the specific versioned directory that is typically referenced.  The idea is to extract the read-only aspects (code) of Tomcat out to read-only NFS storage and the individual web servers have the application specific tomcat configuration files and directories (log, webapps, work).  This can speed up the process of upgrading to a newer Tomcat version - especially for minor upgrades.  I do recommend cleaning out any unused functionality from the Tomcat server.xml file.

If the web servers used above are virtual machines, creating more servers to increase scalability may only require cloning an existing server and making sure it is in the load balanced pool.

These same ideas can be applied to non-production applications and some aspects can be shared between applications (like the read-only NFS storage containing Tomcat and/or JDK).  As the number of applications and servers increases, the benefits from these ideas increase.

And yes, you should only do this if you trust your storage system.  I recommend a backup plan for handling any major infrastructure failures.  Having a copy of the NFS data available for installation on the local servers is one possible method.