Selecting technology for a software system

Selecting technology for a new software system is challenging. There are very diverse technologies available and opinions on each are equally diverse. You may find some people with the opinion that the selection isn't hard because there is only one good choice (or close to it) - in their opinion given their requirements and assumptions.  

Why do I feel the selection is difficult? There are a number of reasons. 

• Is there a perfect programming language for all needs?
• No; each language has trade-offs.
• In a number of cases, a specific language may be a good choice but you also need to consider different runtime environments for a language and the inherent trade-offs.
• Different business domains / uses may match up better to specific languages or technology stacks.
• A language / runtime isn't a solution on its own; you also need to consider the overall environment in which the software will operate.
• Some software systems require higher security and some technology stacks are more mature in that area.
• Some systems are especially sensitive to the speed of implementation.
• The expected lifetime of a system and maintenance needs affects choices.
• The maturity of the 3rd party ecosystem for a language / tech stack affects choices.
• Another item which tends to be considered but usually without thinking about it is - what languages are your staff skilled with?
• And then you have the choices affecting performance, scalability, reliability, cost, etc.

The above list is a bit abstract in nature.  What are some very real and important examples?

A common language that comes up often as the "golden choice" is Python. Is Python a good language - yes, for appropriate uses/domains.  I won't dive into what are good uses for it, I tend to prefer knowing how to rule out items so I can get to a short list for final selection. In the case of Python, what aspects are worth considering? 

• Memory handling
• Threading / Giant lock
• Build time / dependencies
• Backers of the language

One aspect of memory handling that is problematic (at a minimum for CPython) is that there is no control of limits by the runtime system. In standalone / local applications, this isn't a huge burden but if you are using a containerization / orchestration type systems such as Docker or Kubernetes (K8S) then you run into a dilemma.  Docker / K8S enable you to set memory limits and when those limits are exceeded, generally the container is killed and restarted.  The CPython runtime isn't aware when there is memory pressure and isn't tuned to attempt to prevent over-allocation.  To prevent containers from over allocating memory and being killed - more headroom memory is needed per container instance.  This can reduce the efficient use of memory which increases cost.  If you try to run more CPython processes in a container to improve generally efficiency - you can cause more workloads to restart due to a single over-allocation. The result is poor user experience. Increasing the number of container instances requires adequate memory for each which translates into higher costs.  

CPython also has a giant lock which impedes its multi-threading capabilities (Python GIL / Threading). So managing high request rates in a web application may provide some challenges. You may think it is easily resolved by use of some messaging type system (maybe Kafka, MQ, etc) but if the client libraries don't work-around the inherent issue then you end up with problems that may be hard to diagnose and even harder to fix. 

If you are using a standard interpreted version of Python then you don't really need to worry about build time related to your actual code but you may need to deal with the time required for building any required native dependencies. There are many libraries that work with Python but they are implemented using native libraries which the Python runtime loads and uses. This often means you need C/C++ and / or other compilers to enable building the native libs. This can be slow and error prone when working with containers initially. Given time and effort, you can create a solid build process but this is something that brings the overall complexity of creating, containerizing and deploying a software system to levels similar to compiled languages.

Every popular language remains popular by changing enough to meet new needs and challenges in the software development ecosystem. In some cases, languages are backed by committees (C++) and others have a community/large organization as the primary backing (Java, .Net). For Python, Guido appears to maintain primary control and he has done a wonderful job. The question is - someday, when Guido decides to step away completely - who or what controls the direction of the language and will it diverge much from Guido's current direction?  One link regarding the "no future Python 4" path is: No Python 4.

If you invest heavily in Python and something changes significantly (and quickly) then it could be very costly to change. At the same time, if a language looses popularity after its creator moves on then you may also be in a bad state.

The above coverage of Python is mainly targeted at CPython. The analysis for a different runtime, such as Jython, would need an independent review.  Performance, backing/maintenance and implementation details can be very different.

Another common language stack is NodeJS / JavaScript. It is often touted as high performance and has a large ecosystem of 3rd party libraries.  What aspects of this might affect the consideration for some software system?

Two item that comes up regularly for me are

• The maturity of the 3rd party ecosystem. 
• The expected lifetime of a system.

The problem I tend to run into is that there are many 3rd party libraries but the quality differs drastically and the long term support/maintenance of some frameworks is lacking. If you have a system which you expect to maintain for 5-10 years then you probably want to base it on technology which gets continued incremental maintenance over time.  Relying on unmaintained libraries and frameworks when security is important seems like a poor bet. You also don't want to do massive rewrites of a system every year because some new "great idea" arrived.  I've run into situations where core libraries used in NodeJS applications were abandoned by authors in favor of other completely different solutions.  I'd even agree with the authors decisions to abandon something which was no longer a good fit - but it isn't a good place to be in if you heavily rely on that software in a large or complex system.

Another popular language is Java - what are some considerations for it?

• Speed of system implementation
• Business domain for new system

Java is a good general language but it isn't normally considered as the first choice when "speed to market" is the most critical aspect. If you have a very limited time frame and no other requirements push you towards Java then another language may be more appropriate.  That is also true if your business domain is potentially in a scientific area where there are fewer 3rd party libraries available compared to a language such as Python. 

And if you have requirements which cross a number of these items resulting in no "perfect for this use" selection then you have hard decisions to make. 

This post could go on for many more examples and languages but hopefully provides a useful analysis for some of the challenges involved.

Wishing you the best!

Scott

Reasons for poor quality Software Systems

I've seen issues with software system quality over the years - it wasn't just one type of organization or domain. Why is this true even with continued innovation and best practices? I would expect to see improvements over time.  There are plenty of tools to help improve software quality.

Great IDE's

• IntelliJ
• Eclipse
• Visual Studio

Language and compiler improvements

• Java 11-19

Source scanning tools

• PMD
• SonarQube
• FindBugs
• CheckStyle
• SpotBugs
• VeraCode & Twistlock
• more related to security but IMO there is a relationship between security and quality

Code generation tools

• Lombok

Additionally, features like Java annotations are leveraged heavily nowadays which simplify configuration and lower overall code quantity/complexity.

I've found two common aspects across a number of organizations which I think are related to the quality issues.  The first aspect is lack of attention to the warnings listed in common IDE's.  I've seen some applications showing over a thousand warnings when I first worked with them.  This crosses implementation languages/run-times as well - Java, Python, JavaScript, NodeJS, etc. Of course, often IDE's only show the first 100 or so warnings so you don't even have a complete list typically.  

The second aspect is the use of technology, such as general annotations and also Lombok, without paying attention to all the details regarding how each related annotation works (for default and non-default settings).   Default behavior isn't always the required behavior. An example of this is:

create a class such as:

@Data

@AllArgsConstructor

@NoArgsConstructor

public class BaseData

{

/**

* Represent identity; lombok generated equals() should

* utilize this to determine result.

*/

@NonNull

private String baseDataId;

}

 

which is fine on its own.  Note that the name-prefix "Base" easily implies that you may create sub-classes though. Let's do that here;

@Data

@RequiredArgsConstructor

public class DerivedData extends BaseData

{

/**

* Represent identity; lombok generated equals() should

* utilize this to determine result.

*/

@NonNull

private String derivedDataId; 

}

No compile error is generated.  Will this work?  It depends - how are you expecting to use this class?  If you expect that the equals() method will account for both the baseDataId and derivedDataId then you will have an unhappy surprise. 

If you pay attention to warnings in your IDE though you will note a warning is generated. 

Generating equals/hashCode implementation but without a call to superclass, even though this class does not extend java.lang.Object. If this is intentional, add '@EqualsAndHashCode(callSuper=false)' to your type.

I've not seen PMD, Sonar, etc complain about this so those tools leave you feeling like "everything is awesome".  

Add some unit tests which exercise the equals() method in the situation where you are using the DerivedData class and have instances with matching derivedDataId values but different values for baseDataId.  You'll probably write the test indicating the instances should be non-equal.  Will you be surprised when the tests you setup indicating instances should be non-equal actually fail?

The reason it fails is that the generated equals() in the DerivedData class doesn't know anything about the members of the BaseData class.  So equals() in the derived class only uses the derivedDataId member. 

If you review the warnings in your IDE, you will find that it implies how to resolve this if it wasn't intentional - add annotation '@EqualsAndHashCode(callSuper=true)'.  But notice I said "it implies" because it directly tells you how to ignore the warning about this default behavior.  So a good question is - how often is the default behavior correct versus the logic always calling superclass equal() / hashCode()?  I suspect that the current default is opposite from what would be a safer default.

And this brings us back to IDE warnings being ignored.  If you ignore / hide those warnings you easily end up missing these types of errors which can show up as "anomalous production issues" at some point.

I think that an aspect of all this is the attempt to (over)simplify everything. So many details are abstracted away that developers won't see implications of all the "defaults" and "configuration by exception", etc that occurs is so many systems.

Thanks for reading..

Scott

Random Recent IT "stuff"

General issues

• Ubuntu 21.04 upgrade
• sizing tmpfs - default 36G on my system seems excessive; what was eating up memory/swap the other day
  
• Rootless Docker push to Microk8s Registry
  
•  Get https://docker-star-cases:32000/v2/: dial tcp: lookup docker-star-cases on <x.y.z.q>:53: server misbehaving
• What about podman, buildah, etc? 
  
• Microk8s 1.21.0 upgrade
• kubelite - somewhat excessive CPU?
• DNS
• Still working out some DNS ideas
• journal 
• logging tuning 
• Routing / firewall tuning
  

 

Other known challenges

• sorting out my nvme and ssd partitions and best usage
•  Careful not to conflict with snap or normal ubuntu upgrades

Potential plans

• Upgrade to 128G RAM
• Allow creation of some specific ramdisks to minimize wear on nvme/ssd drives.
  
• Upgrade to i9 processor
• More cores/threads to spread the services over.
  
• Commercial or self-built NAS
• Simplify data management and provide high-speed access to more systems.
• UI work

In-progress / todo

• setup bitnami/openldap in Microk8s
• enable tls / cert rotation
• some oauth2 related selections
• App integration with geoserver 
• Catch-up Graalvm version of services with SpringBoot version
• 
  

Application Server Deployment

What are drivers of an organizations deployment strategy? 
 The ones that come to mind are

1. Specific technology investments such as WebLogic (think of cluster deployments).
2. Generic process based on common technology (think of Tomcat Manager)
3. Custom processes, infrastructure and configuration to meet specific needs

Which is appropriate? I think the answer is "It depends".

If an organization has an investment in WebLogic or other costly technology and the deployment capabilities meet it's needs then maybe there isn't a need to do anything different than the product dictated solution. Using WebLogic or similar solution likely means there are multiple tiers and maybe load balancing of one or more tiers over multiple servers.  That type of environment is substantially more complex to setup and maintain than a single tier/server solution.  If the organization is staffed appropriately and has the proper training with the specific tools then doing something different seems unwise unless there is a good reason.  The main downside to this solution is cost - for both the tool and either training or hiring of the appropriate staff.  If the application hosted is critical then the costs multiply as well since you likely would want multiple staff trained to provide coverage during vacations or other unplanned emergencies.

Many small/mid-size organizations likely use Apache Tomcat or something similar.  There are reasonably well documented install and deployment procedures available on-line or in various books.  For organizations with smaller or less trained IT departments this is certainly a workable solution.  Most developers are more than capable of getting a default install up and running and managing it.  Applications can be scaled by load balancing multiple servers which starts to increase the complexity but is likely manageable at smaller scales.  After a certain point, scaling via adding more load balanced servers and using the default deployment procedures starts to become fragile.  Examples of the result of the fragility might include failed deployments, individual servers return wrong results, servers accessing wrong resources or what can be best described as flaky behavior.  These problems likely stem from things such as improper change management, rushed planning, lack of reliable deployment tooling/procedures, tight maintenance windows, etc.

Are there alternatives to the above and why would an organization use them?  Yes there are alternatives to the above scenarios and I will document one here.  This is only an alternative and is not necessarily the best one for every organization.  Each and every organization has to evaluate their resources, risks and requirements to determine what is appropriate for themselves.  This alternative focuses on reducing deployment errors, speeding up the deployment process for some cases and some other benefits I will document.

The assumptions for this alternative are:

1. mid-size organization
2. understaffed IT with wide duties
3. critical applications with at least a moderate rate of change
4. basic web application (multiple load balanced web servers and 1 DB server)

The basic technology used for this particular solution are:

• NAS NFS storage
• Apache Tomcat
• Linux based Web servers

Since many deployment errors can be traced back to mistakes updating configuration data manually across multiple servers.  The basic solution is to externalize the primary configuration data to a location outside of the web application where it resides on NFS storage.  Primary configuration data means data which differs between the production and non-production environments - such as DB connection/pooling info, special URL's, etc.  By moving the data outside of the application and web server and storing in a single location you reduce the touch points during future deployments.  If the primary configuration doesn't need to change then you don't need to edit those files during normal deployments.  This reduces the risk of unintended changes.  This can also speed up deployments since you may only require a restart of the remaining web server processes once the shared configuration change is made.

Another improvement to the overall environment includes installing the Java JDK on NFS storage and having a Linux soft-link (in a location on the NFS storage) to the current JDK in use.  Any later JDK upgrade process is faster because you only need to do one install, change one soft-link and start the application up.  These last 2 steps are the only steps where the application should probably be down.  If you have a lot of servers to deploy to, this can be a real time saver.  This also saves some space but storage is relatively cheap so the benefit is likely minimal.

Another idea is to install the Apache Tomcat software on NFS using the delivered ability to specify CATALINA_BASE and CATALINA_HOME separately.  After doing this, setup a Linux soft-link (in a location on the NFS storage) to the current Tomcat.  Each web server would reference the soft-link instead of the specific versioned directory that is typically referenced.  The idea is to extract the read-only aspects (code) of Tomcat out to read-only NFS storage and the individual web servers have the application specific tomcat configuration files and directories (log, webapps, work).  This can speed up the process of upgrading to a newer Tomcat version - especially for minor upgrades.  I do recommend cleaning out any unused functionality from the Tomcat server.xml file.

If the web servers used above are virtual machines, creating more servers to increase scalability may only require cloning an existing server and making sure it is in the load balanced pool.

These same ideas can be applied to non-production applications and some aspects can be shared between applications (like the read-only NFS storage containing Tomcat and/or JDK).  As the number of applications and servers increases, the benefits from these ideas increase.

And yes, you should only do this if you trust your storage system.  I recommend a backup plan for handling any major infrastructure failures.  Having a copy of the NFS data available for installation on the local servers is one possible method.