Early Thoughts: Quarkus / GraalVM

Containerization has become very important in recent years.  Starting about 4 years ago, jobs I've had went from no containerization in local data centers to running dockerized images in a cloud such as AWS or Azure. Initially this was just using docker directly on a cloud server.  Over time, the work has gradually migrated more towards Kubernetes though.

There are a lot of lessons learned and a few still being learned. One area of continued learning is in the area of cost. It is so easy to spend big $ in any cloud environment. The choices we make in the tech stack and architecrual areas has a huge impact on final costs.  

Native serverless solutions can become cost prohibitive when the call rate is extremely high. Slow virtual server startups means you must over-provision more initially to prevent lag time issues with auto-scaling. Fat dockerized images eat up memory typically requiring larger underlying hardware to provide additinal memory.  High initial docker container processing results in further over-provisioning to compensate for lag-time. Choices in runtime/language affect memory requirements as well.

With that said, I've been starting to toy with some relatively new technologies in the bits of spare time I have at home - Quarkus (https://quarkus.io/) which also integrates with GraalVM (https://www.graalvm.org/).

The ability to create an optimized container from Java microservice code which prunes out unused functionality and converts to a native application has some benefits.  The biggest touted benefit appears to be lower memory requirements for the deployed container and startup speed but I also think it might help improve security overall - needs some research.

My initial work with Quarkus (non-native, just local images) has been pretty straight forward. I am able to work with it from Eclipse without significant issues.  I wil say that memory requirments tend be excessive for the build process (especially native application image). Without limiting memory, it consumed too much and caused system instability and trying to tweak the memory setting for the build itself is a bit of art.  Here is a snipped of the out of memory failure with 5Gb dedicated.

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM Version 19.3.1 CE[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep]...
[notesvc-quarkus-1.0.0-SNAPSHOT-runner:24] Exception in thread "ForkJoinPool-2-worker-9" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
Fatal error: Exception in thread "main" java.lang.OutOfMemoryError: Java heap spaceError: Image build request failed with exit status 1

Once I bumped the setting in the POM file to 6Gb and reran build it succeeded.  Here is the profile that worked:

   <profile>

      <id>native</id>

      <activation>

        <property>

          <name>native</name>

        </property>

      </activation>   

      <build>

        <plugins>

          <plugin>

            <artifactId>maven-failsafe-plugin</artifactId>

            <version>${surefire-plugin.version}</version>

            <executions>

              <execution>

                <goals>

                  <goal>integration-test</goal>

                  <goal>verify</goal>

                </goals>

                <configuration>

                  <systemProperties>

                    <native.image.path>${project.build.directory}/${project.build.finalName}-runner</native.image.path>

                  </systemProperties>

                </configuration>

              </execution>

            </executions>

          </plugin>

        </plugins>

      </build>

      <properties>

        <quarkus.package.type>native</quarkus.package.type>

        <quarkus.native.additional-build-args>-J-Xmx6G</quarkus.native.additional-build-args>

      </properties>

    </profile>

The startup time does appear significantly faster on first tests.  I'll come back and add some of those details soon (hopefully).

Hope your day is blessed!
Scott

  

Early thoughts: MicroK8s

For now, it appears that Kubernetes is the default method for orchestrating containers in many organizations. Since my software development work has revolved around enterprise cloud solutions for the last few years, I've gone through various proof of concepts using other technologies but in the end - kubernetes tends to end up as the preference by many organizations.

One area which I've not seen handled really well is the difference in working "microservices" at the individual developer level (local sandbox) vs in a shared development environment (cloud PaaS K8S). This also related to the initial bootstrapping of new services into a cloud/K8S environment. 

Having per developer resources in a cloud can be cost prohibitive at scale and sharing resources results in coordination issues or worse (for early development).  Most places don't seem to have a standard development method across the board for local development - which shows up as bugs in early deployments.  Some developers are using Docker by itself to manage private resources or some combination with that and very specific shared resources (maybe a dev DB).  Other developers might be using a Docker/K8S solution on a Mac. So far, it often seems awkward at scale with few good/standardized practices.

With that in mind, at home I started working with Microk8s (https://microk8s.io/). I'm hoping to derive some better practices/processes which can be shared. There are a few K8S solutions I could work with but MicroK8S was something I ran across pretty early on.

Overall it works ok - there are differences between PaaS cloud K8S, role your own K8S on cloud servers and using Microk8s on a single physical system but it is still very useful from both functional and learning perspectives. I did a 'snap' install of MicroK8S on Ubuntu and after a bit fiddling got it working.  I have had a few issues when trying to do an upgrade but we'll see how things work out over time - maybe I should use a different installation method.

It does consume resources but using an i7 based system with 16GB RAM, I am able to run MicroK8S, docker, Eclipse, Chrome, DBeaver and and a few dockerized contaniners (MySQL, Postgres, etc).   I'm trying to limit the addons for now to just a handful which are focused on metrics,monitoring,ingress,storage,registry, dashboard and Istio.

I should create some aliases for commands because I do find it tiresome typing "microk8s kubectl ..." or the other dozen or so commands.

It is easy to start/stop MicroK8S so if I find I need to do other work and need the resources, I can usually just stop it.

When I started to combine other tech into the stack with this; memory has become an issue.  Using Quarkus fo build native images is super resource intensive - I'm considing upgrading to 64GB ram even it I need a new system to do so. I'll leave those details for a different post though.

Application Server Deployment

What are drivers of an organizations deployment strategy? 
 The ones that come to mind are

1. Specific technology investments such as WebLogic (think of cluster deployments).
2. Generic process based on common technology (think of Tomcat Manager)
3. Custom processes, infrastructure and configuration to meet specific needs

Which is appropriate? I think the answer is "It depends".

If an organization has an investment in WebLogic or other costly technology and the deployment capabilities meet it's needs then maybe there isn't a need to do anything different than the product dictated solution. Using WebLogic or similar solution likely means there are multiple tiers and maybe load balancing of one or more tiers over multiple servers.  That type of environment is substantially more complex to setup and maintain than a single tier/server solution.  If the organization is staffed appropriately and has the proper training with the specific tools then doing something different seems unwise unless there is a good reason.  The main downside to this solution is cost - for both the tool and either training or hiring of the appropriate staff.  If the application hosted is critical then the costs multiply as well since you likely would want multiple staff trained to provide coverage during vacations or other unplanned emergencies.

Many small/mid-size organizations likely use Apache Tomcat or something similar.  There are reasonably well documented install and deployment procedures available on-line or in various books.  For organizations with smaller or less trained IT departments this is certainly a workable solution.  Most developers are more than capable of getting a default install up and running and managing it.  Applications can be scaled by load balancing multiple servers which starts to increase the complexity but is likely manageable at smaller scales.  After a certain point, scaling via adding more load balanced servers and using the default deployment procedures starts to become fragile.  Examples of the result of the fragility might include failed deployments, individual servers return wrong results, servers accessing wrong resources or what can be best described as flaky behavior.  These problems likely stem from things such as improper change management, rushed planning, lack of reliable deployment tooling/procedures, tight maintenance windows, etc.

Are there alternatives to the above and why would an organization use them?  Yes there are alternatives to the above scenarios and I will document one here.  This is only an alternative and is not necessarily the best one for every organization.  Each and every organization has to evaluate their resources, risks and requirements to determine what is appropriate for themselves.  This alternative focuses on reducing deployment errors, speeding up the deployment process for some cases and some other benefits I will document.

The assumptions for this alternative are:

1. mid-size organization
2. understaffed IT with wide duties
3. critical applications with at least a moderate rate of change
4. basic web application (multiple load balanced web servers and 1 DB server)

The basic technology used for this particular solution are:

• NAS NFS storage
• Apache Tomcat
• Linux based Web servers

Since many deployment errors can be traced back to mistakes updating configuration data manually across multiple servers.  The basic solution is to externalize the primary configuration data to a location outside of the web application where it resides on NFS storage.  Primary configuration data means data which differs between the production and non-production environments - such as DB connection/pooling info, special URL's, etc.  By moving the data outside of the application and web server and storing in a single location you reduce the touch points during future deployments.  If the primary configuration doesn't need to change then you don't need to edit those files during normal deployments.  This reduces the risk of unintended changes.  This can also speed up deployments since you may only require a restart of the remaining web server processes once the shared configuration change is made.

Another improvement to the overall environment includes installing the Java JDK on NFS storage and having a Linux soft-link (in a location on the NFS storage) to the current JDK in use.  Any later JDK upgrade process is faster because you only need to do one install, change one soft-link and start the application up.  These last 2 steps are the only steps where the application should probably be down.  If you have a lot of servers to deploy to, this can be a real time saver.  This also saves some space but storage is relatively cheap so the benefit is likely minimal.

Another idea is to install the Apache Tomcat software on NFS using the delivered ability to specify CATALINA_BASE and CATALINA_HOME separately.  After doing this, setup a Linux soft-link (in a location on the NFS storage) to the current Tomcat.  Each web server would reference the soft-link instead of the specific versioned directory that is typically referenced.  The idea is to extract the read-only aspects (code) of Tomcat out to read-only NFS storage and the individual web servers have the application specific tomcat configuration files and directories (log, webapps, work).  This can speed up the process of upgrading to a newer Tomcat version - especially for minor upgrades.  I do recommend cleaning out any unused functionality from the Tomcat server.xml file.

If the web servers used above are virtual machines, creating more servers to increase scalability may only require cloning an existing server and making sure it is in the load balanced pool.

These same ideas can be applied to non-production applications and some aspects can be shared between applications (like the read-only NFS storage containing Tomcat and/or JDK).  As the number of applications and servers increases, the benefits from these ideas increase.

And yes, you should only do this if you trust your storage system.  I recommend a backup plan for handling any major infrastructure failures.  Having a copy of the NFS data available for installation on the local servers is one possible method.