JSR 376 / Java Platform Module system and Service Loader experience

 I have some pet projects I work on and I decided to convert one from a plain Java project over to using the Java Service loader and the Java Platform Module System.  The project uses a few well known libraries but isn't using Spring or other giant frameworks. I figured this should be pretty straight forward to do.

I started by converting over to the Java Service loader and after a bit of refactoring I got that working pretty well.  I expect anyone trying to convert an application or service over to it will find plenty of things that should be refactored to make it work well. It was a good experience though even though it was a good amount of work initially.

And then I started the conversion to the Java Platform Module system.  This was pretty difficult to get started and required more work and refactoring.  Some issues may simply be tooling issues in Eclipse or something along those lines.  For my first attempt, I eventually bailed on it and started from scratch again. 

At this point, I did make lots of good progress. I had things pretty much sorted out except for one thing. I had a problem with the Infinispan cache and the module system.  I didn't expect that and I tried all kinds of things to get past the issue.  I eventually ended using the JSR 107 Cache-api to abstract as much as possible from Infinispan specifics.  At this point, I expected it to work but I finally ended up at a particular error. The error indicated that there was an Infinispan package name that existed in two different jar files (with different classes) and that isn't supported by the module system.  I thought maybe it was a regression or small oversight - I mean, the module system has been around for some time now.  

I did a search of Infinispan issues for JSR 376, Jigsaw and platform module system but found no issues.  I finally put in an issue and the next day the tech lead closed the issue as a duplicate.  The original issue was reported by that tech lead back in 2019 as a high priority ticket.  If I had searched for "module", I probably would have found it.  Anyways, being that this was known since 2019 and the original ticket mentioned more packages that were split between jars - I am guessing this won't end up fixed anytime soon.

I removed Infinispan and am just using the JSR-107 support as a way to hide my use of a simple map for in-memory use for now.  I wanted to persist my data (which was a key benefit of Infinispan) so I briefly looked at creating my own file based persistence layer using some plain IO stream support along with protobuffers but even that had some issues.  

I'm planning to switch to ASN.1 - that is more standardized and has better "enterprise support" than most of the other serialization frameworks. Try finding active open source projects for "serialization" from google results related to that.  There are a few but there are many more that seem pretty dead. ASN.1 isn't going away anytime soon and there is an active Java project supporting it (along with many other language projects).

Thanks for reading!

Scott

[2023/06/11 Update] I'm coming to the conclusion that using the Java Platform Module system is both too high of a risk and too painful to do for any large software system. It is a risk because you can't guarantee it is going to work in a fully modular system because so many libraries and transitive dependencies are not modularized and/or have split packages. Based on how long the feature has been around, I don't see the situation improving without some major push that forces it. Trying to use Maven plugins such as "moditect-maven-plugin" and the "jlink" tool is way more effort than a short or mid-term payback would make worthwhile. Working with this tooling and trying to resolve the issues one at a time is a huge headache - certain errors are not found in a deterministic fashion. This means that you add a "fix" into the config and rerun your build and you don't get the error but you end up with a different one.  This doesn't mean you fixed the original error, it just means that it didn't pick the same problem point to report on.

I'm also rethinking the ASN.1 conversion. I was able to get the asn1bean project compiled (i.e. the ASN1 Compiler part) with Java 18 with some effort. I'm still determining if the generated code will really work the way I need/expect it to.  I've got some other things I want to work on now so maybe I'll come back to it later.

Application Architecture and Design (and cloud aspects)

 Let's say you work for some organization and there is a need for a new application.  If you are responsible for the architecture / design then 

My initial post was going to end up being a "ancient" IT history book with some extra stuff. I decided that wasn't very helpful - not many people want to read a book in a "blog format".

I'm condensing the content down to the following. 

• Use standard practices for
• project management
• design techniques
• Understand the reasons/basis for soft/hard requirements - This covers why a requirement exist and why a specific requirement was chosen over other similar prospective requirements.
• Examples reasons are;
• development / operational / infrastructure costs
• time to market /  speed of handling requirement changes
• maintainability
• performance / scalability / availability
• Carefully evaluate technology choices.
• Consider the risk of using any technology that could become unsupported before the end of the useful life of your application. New "amazing" technologies may only be around for a short period of time before something new replaces it.
• Make plans to keep 3rd party frameworks, libraries and languages/run-times up-to-date with regard to the most current versions.
• Consider the risk, for example, of using early versions of SpringBoot such as 2.1 which are not maintained now - if a new security flaw comes up, it may be very hard to fix the issue.

With regard to standard practices, use whatever is appropriate for your organization and the project need. Many organizations are trying follow "Agile" practices but that doesn't mean that "Waterfall" is always bad to use. Use what is appropriate for the situation.

Understanding the reasons used for requirement selection helps to keep requirements aligned with each other and provides useful context for handling business changes over time.

Technology comes and goes.  Don't love or hate particular technology - it is a tool.  Use appropriate technology and if needs change then consider more appropriate technology.  Don't select technology just because it is considered "great" by any person or group - consider the positive and negative aspects in-depth for your use-case.  

Small example of considerations - given a service expecting high transaction rates; what technology stacks might be considered and what are the trade-offs?

1. Python
1. Positives
1. Increasingly used in enterprises
2. OpenSource
2. Negatives
1. Library ecosystem isn't as mature as for Java
1. You want a library to help with web requests?  Which option do you pick? What version of Python are you on?
1. https://pypi.org/search/?q=web+requests&page=1
3. Considerations
1. Security
1. https://www.theregister.com/2021/07/28/python_pypi_security/
2. https://medium.com/ochrona/arbitrary-code-execution-during-python-package-installation-3a60990350ef
3. https://snyk.io/blog/python-security-best-practices-cheat-sheet/
2. Some libraries need local compilation
3. Cross-version compatibility
4. Transitive dependency handling
5. Long term maintenance questions
1. https://medium.com/codex/python-4-0-will-never-arrive-3d994dce54f1
6. Runtime aspects
1. Python global lock
1. https://wiki.python.org/moin/GlobalInterpreterLock
2. Memory management - also consider context including containers
1. https://www.askpython.com/python/examples/memory-management-in-python
3. Threading
1. https://docs.python.org/3/library/threading.html
2. Java
1. Positives
1. Heavily used in enterprises
2. Generally has useful features and reasonably performance
3. Many enterprise level libraries and frameworks
2. Negatives
1. New version every 6 months by Oracle can cause some thrashing of work
2. OpenSource vs Oracle licensing
3. Considerations
1. Alternative JDK / JVM / run-times
1. Oracle JRE
2. OpenJDK
3. Azul
4. IBM OpenJ9
5. Eclipse Temurin
6. Amazon Corretto
2. SpringBoot vs JEE
3. Cross-version compatibility
4. Transitive dependency handling

I highly recommend reading about and understanding the runtime aspects of Python. Consider the effects of memory pressure within and across container instances (i.e. Docker, Kubernetes, etc) - especially in a cloud environment and / or where you are cost / latency sensitive. 

Regarding compatibility and transitive dependency handling - I've had people with architect and team lead titles make comments to me along the lines of:

• "Java versions are backward compatible so keeping up with current versions isn't important."
• "We are pinning the SpringBoot version at version 2.3 instead of upgrading to 2.7+ because it improves stability by preventing changes."

I believe these statements reflect some general misunderstandings or lack of knowledge. There are very instances of 100% cross-version compatibility for operating systems or programming languages over significant spans of time. You also have to clarify whether you are talking about things like source code / binary compatibility and forward / backward version compatibility. Try running a bunch of games written for Windows 95 on Windows 11 - how many will work?  In my opinion - likely few.  For operating systems, computer languages and frameworks - features are added and deprecated / removed across releases.  API's change due to needs such as performance, security, new features, etc. If OpenSource is brought up as a solution to these type of issues - I'll claim it isn't reasonable.  There are support matrices for most major OSs, computer languages and library / frameworks - they don't maintain old versions forever. If you are outside a support matrix - you have no guarantee that anything will work and getting help will likely be somewhere between hard and impossible. Needing help for a major security issue or something that breaks a critical system is not the time you want to find out that no one can help you or can't help you quickly. Some common support / compatibility matrices below..

• OpenJDK support matrix from RedHat
• https://access.redhat.com/articles/1299013
• SpringBoot support matrix
• https://spring.io/projects/spring-boot#support
• SpringBoot / SpringCloud compatibility matrix
• https://spring.io/projects/spring-cloud#overview
• Django support matrix / roadmap
• https://www.djangoproject.com/download/
• Nx / NodeJS / TypeScript compatibility matrix
• https://nx.dev/packages/workspace/documents/nx-nodejs-typescript-version-matrix

Thanks,

Scott

PrimeToolkit - BETA

I've had a pet project for a while related to Prime Numbers and a number of other things.  I finally took some time and created the start of what i'll describe as a toolkit.  

I've made it available on github; https://github.com/scottcase3374/primetoolkit

[Update 2022/01/25]

I'm integrating in the LWJGL library to allow creating some different 3D representations of some of the data. Hopefully that won't take too long to get into a useful state.

The LWJGL libraries were a bit excessive considering some of my goals.  I've still got the default visualization support I started with and have cleaned up and extended things quite a bit.

I exported some of the data to visualize it using an opensource program- OpenSCAD.  See my blog entry about the visualized data.

The toolkit is organized along the lines of this:

The above diagram was created using yed. The source diagram file is found here.

I'm fairly happy with how it has turned out so far but with software development - it may never be completely "done" as I find new things to experiment with or tweak.  I'd still like to include some metrics generation and a few experimental data structures to compare time/space trade-offs.

Below is a quick demo of it - the demo is not as polished as desired but will hopefully be understandable enough for now.  I did end up reducing some of the data ranges for the demo since running the demo and OBS studio at the same time was fairly resource intensive.

Thanks,

Scott

Graphing Prime Numbers

I enjoy toying with prime numbers and wanted to see how the Java GraphStream library worked.  Here is a short video of the result. 

 

 

Interestingly, it is little fun projects like this that often turn into moments of learning.  While working on some prime number related programs I learned of some tool limits I wasn't aware of before this.

•  The max radix value for converting numbers to strings is 36.  From some searching, it sounds like that was chosen by being the number of values 0-9 + the 26 case-insensitive values a-z.  I found it odd when numbers I converted starting at radix 37 were identical.
• Integer.toString(99999, 37)
  
•  For a call to: com.google.common.collect.Sets.powerSet( aSet) , the "aSet" set parameter is limited to a max of 30 items.

 

 

I decided to try a variety of layout parameters to look for interesting results.  Here are a few of those results using GraphStream.

 

 Those were interesting but I decided to see if I could integrate different graphing libraries to look for other interesting features, etc.

My first attempt was to try integrating gephi but that didn't go well.   As I looked into other graphing libraries, I noted that many are not maintained.  This took way too much effort due to the version of netbeans libraries that had been used.  I ended up forcing newer versions which are hosted by Apache.  Eventually, I had code that built but no visual graph was produced - I noted some flakiness with one of my displays when trying to run it so I suspect an issue between older code and my display/drivers.  

Next, I decided to try Jgrapht.  This had an initial snag which was the fact that the original package was no longer maintained and wasn't available via maven.  I ended up finding a version that is either a fork or repackaging of it and is available via maven.  Now, things were much less difficult.  It also resulted in very few total dependencies - even after I reworked the code significantly and added in the picocli lib for command line arg handling. I also ended up needing to add in the jgrapht-ext and jgraphx libs to display resulting graphs.

Here are a couple examples from that. The first uses the pre-canned compact tree layout. It took some playing to get something useful to display though. The second graph uses the pre-canned circle layout and that was easy to setup but still not as visually appealing as I desired.

I'll keep looking and trying new settings to see if I can come up with something better but this does work at least.  From a visual graph standpoint, it doesn't scale very well it seems but maybe some tuning is possible.  I got the generation of the primes tuned much better now and could generate 1M primes quickly but trying to graph even 10k primes seems unrealistic / painful.  I'll probably end up terminating the running process before it ever gets to the point of displaying the graph.

[update] I added the ability to export my data into the Graph Modelling Language (GML) format and that is compatible with the yed tool. Yed has much richer layout support that what I have been working with from a Java library integration standpoint.  The downside is that really large datasets are time consuming to visualize.  I'm not sure how long it took for yed to complete this since I took a several hour break.  This was for 10k primes.

SpringBoot Batch load CSV into Postgresql - external with K8s configmap

 This is a short demo of running a SpringBoot batch job in eclipse where the configuration is retrieved from a Microk8s configmap.  The setup can be extended to use K8s secrets but wasn't done here yet. 

I had worked with several other tools when initially trying to import this data into my database but I ran into a number of problems with the methods.  The biggest issue is some "complex" data as part of a set of fields that most import tools/processes don't handle correctly but where handled here fine.  It did take a few iterations to work out a few incorrect field data types but Spring Batch provides pretty clear information regarding the problematic fields and types.

 

 

Here you can see an example of what wasn't handled well by some other import methods. It usually included partially quoted data and also commas.

 Here is a quick screen shot of the Microk8s info showing my Postgresql database. My batch job uses the service name as the host portion of the connection string. For now, I manually update my DNS resolutions to map the service name to the service IP from outside of my cluster. I'm working toward that being an integrated/automated item requiring no manual intervention.

Thanks for looking and have a blessed day!

Scott

 

Local Development, Microk8s Built-in registry and DNS Parity - Part 3

In part 2 of this, I had a SpringBoot service which was runnable from both an IDE and from Microk8s - with the configuration for both methods coming from a configmap in Microkk8s.

Here I want to go into the configuration a bit more because that part created a few headaches but I think it is sorted out now (for the most part).

To start with, one of the major enablers for this setup is a bootstrap.yml file.  In this case, I'm not using any application.properties file in the apps main/resources folder.  The use of the bootstrap file is to get required settings, well - bootstrapped before the majority of Spring auto-configuration starts up.

 Spring profiles are also a key part of this setup. The main spring config in the bootstrap file is defined for the profiles:

• native
• default
• kubernetes

 I'll admit that I should have documented this better initially - a key aspect is the kubernetes profile though. There are likely other ways to get things working but so far this is cleanest and most reliable. Note that the kubernetes profile is used even when executing the service in the IDE.  I'll explain that a bit more in minute but first lets look at the initial part of the bootstrap configuration.

 spring:
  config:
    activate:
      on-profile:
      - native
      - default
      - kubernetes
  application:
      name: sb-note
  cloud:
    discovery:
      enabled: true
    kubernetes:
      enabled: true    
      config:
        enabled: true
        enable-api: true
        paths:
        - /deployments/config/application.properties
      discovery:
        enabled: true
      client:
        namespace: default
        clientCertFile: /var/snap/microk8s/current/certs/kubelet.crt
        caCertFile: /var/snap/microk8s/current/certs/kubelet.crt
        ca-cert-file: /var/snap/microk8s/current/certs/kubelet.crt
        trust-certs: true
        masterUrl: https://kubernetes:443
springdoc:
  swagger-ui.path: /swagger-ui.html
debug: true

 

Note that the various cloud/kubernetes aspects are enabled.  These are important for the dependencies:

• org.springframework.cloud:spring-cloud-starter-bootstrap
  
• org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8-config
•  org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8
•  org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8-all

I may be able to simplify those a bit but that is what I have after thrashing a bit with a recent SpringCloud update. The combination provides the ability to "know about" kubernetes and use features like configmaps.  Since I always want to pull my main configuration using a configmap, I always use the kubernetes profile and the enablements above so configmaps are usable.  

Note that the "client" section is what generally provides any configuration needed so your app can leverage the "kubernetes" awareness and access those associated resources. Note that the client.masterUrl though is defined in the manner which the service needs it when it actually is running in Microk8s.  The https://kubernetes:443 is not accessible from outside of Microk8s.  So this config is all I needed for running in Kubernetes - but what about when running it from outside?

Well, there is a second part to the above bootstrap.yml file which is:

---
spring:
  config:
    activate:
      on-profile:
      - ide
  cloud:
    kubernetes:
      client:
        masterUrl: https://kubernetes.default.svc:16443

and when I run the service in the IDE, I add an additional spring profile to end of my list of active profiles: ide

The result is that the client.masterUrl of https://kubernetes.default.svc:16443 overrides the previous definition due to ordering. The kubernetes.default.svc is what I have been manually maintaining an IP for in my home routers DNS.  That DNS name resolves to the cluster IP of the kubernetes cluster and the port 16443 which is what Microk8s exposes it on - see their documentation on ports. 

So this allows the service to see and access kubernetes and the associated resources from both inside and outside of the cluster. Note that I may be able to move some of the other client settings to the "ide" profile but haven't gotten quite that far in my cleanup. I believe that this can be done because I don't think the various paths for the client section are probably visible as shows show - so are probably ignored when running inside kubernetes. 

Note, initially I had things working in a somewhat awkward fashion which this setup fixes. If you don't override the masterUrl then you can still make things work but it is odd.  Basically, if your deployment exposes the configmap as both a configmap AND an application.properties file - when running in kubernetes and the masterUrl doesn't resolve correctly, the configmap won't be found but the application.properties file is mounted and loaded via normal spring auto-configuration.  You'll see some error/warnings in logs but configuration is available.  Once I realized that is what I had done, I worked out this additional profile so to prevent that configuration from occurring.   The current config shown above doesn't result in those error conditions.  

 Note that the configmap is defined as the following.

namespace: default
data:
  application.properties: |-
    spring.profiles.active=kubernetes
    debug=false
    spring.application=sb-note
    spring.datasource.url=jdbc:postgresql://springboot-note-db1-svc.default.svc:5944/postgres
    spring.datasource.username=postgres
    spring.datasource.password=PASSWORD
    spring.jpa.database=POSTGRESQL
    spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
    spring.jpa.hibernate.ddl-auto=update
    springdoc.swagger-ui.path=/swagger-ui.html
    logging.level.org.hibernate.SQL=DEBUG
    logging.level.org.hibernate.type=WARN
    spring.jpa.show-sql=true
    spring.jpa.properties.hibernate.format_sql=true
    javax.persistence.schema-generation.database.action=update

So the configmap data must be available to the service in some form or the database connection initialization will fail.  That is the one of the most visible ways you will know that it isn't finding the information - which is true while running in the IDE and in k8s as well.   I'll note that I am not using k8s secrets here yet - that would be the next step in doing a more "enterprise" setup. That should work in a similar fashion to the configmap setup.

 Note that so far, all this setup is for actual runtime and not deployment - although the above configmap is defined in the way that Jkube needs it so as to create the actual configmap during deployment.  

 For build/deployment, a few key properties are needed in the maven pom file.

        <java.version>11</java.version>
        <docker.starcases.registry>docker-star-cases.starcases.com:32000</docker.starcases.registry>
        <image.user>starcases</image.user>
        <spring-cloud.version>2020.0.1</spring-cloud.version>

        <jkube.masterUrl>https://kubernetes.default.svc:16443</jkube.masterUrl>
        <jkube.trustStoreFile>/var/snap/microk8s/current/certs/kubelet.crt</jkube.trustStoreFile>
        <jkube.version>1.1.1</jkube.version>
        <jkube.recreate>true</jkube.recreate>
        <jkube.deploy.ignoreServices>true</jkube.deploy.ignoreServices>
        <jkube.generator.name>${docker.starcases.registry}/${image.user}/${project.artifactId}:${project.version}</jkube.generator.name>
        <jkube-revision-history>1</jkube-revision-history>

 Note that I am having springboot build the service and JKube is repackaging into a docker compatible image and then deploying that image to the Microk8s registry and then deploying the other resources Jkube generates configuration for from my setup.

The main build is being done using the recent SpringBoot 2.3 feature enhancements - the pom.xml part of interest is.

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <excludeDevtools>false</excludeDevtools>
                    <release>11</release>
                    <image>
                        <name>${jkube.generator.name}</name>
                    </image>
                    <layers>
                        <enabled>true</enabled>
                        <includeLayerTools>true</includeLayerTools>
                    </layers>
                </configuration>
            </plugin>
        </plugins>

 And then rest of the repackaging and deployment is handled by:

        <profile>
            <id>kubernetes</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.eclipse.jkube</groupId>
                        <artifactId>kubernetes-maven-plugin</artifactId>
                        <version>${jkube.version}</version>

                        <configuration>
                            <profile>minimal</profile>
                            <buildStrategy>docker</buildStrategy>
                            <detail>true</detail>
                            <imagePullPolicy>Always</imagePullPolicy>

                            <logStdout>true</logStdout>
                            <verbose>true</verbose>

                            <failOnNoKubernetesJson>true</failOnNoKubernetesJson>
                            <failOnValidationError>true</failOnValidationError>

                            <skipExtendedAuth>true</skipExtendedAuth>
                            <pushRegistry>${docker.starcases.registry}</pushRegistry>

                            <skipBuild>true</skipBuild>
                            <kubernetesManifest>${basedir}/target/classes/META-INF/jkube/kubernetes.yml</kubernetesManifest>

                            <resources>
                                <labels>
                                    <all>
                                        <property>
                                            <name>app</name>
                                            <value>${project.artifactId}</value>
                                        </property>

                                        <property>
                                            <name>provider</name>
                                            <value>jkube</value>
                                        </property>
                                        <property>
                                            <name>group</name>
                                            <value>${project.groupId}</value>
                                        </property>
                                        <property>
                                            <name>version</name>
                                            <value>${project.version}</value>
                                        </property>

                                    </all>
                                </labels>
                            </resources>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>

There are likely a few items I can cleanup here but it is functional.  

 Note that the maven goal I use in the IDE for the Microk8s build / deployment is:

• clean  spring-boot:build-image  k8s:resource  k8s:push  k8s:apply

And I specify a profile of "kubernetes".  I do have a few environmental settings as well - maven.test.skip, skip tests, and an item for trusting the kubernetes certificate which I may not need now (at one point I worked out some issues related to it but am noticing I didn't clean all settings in all the IDE launch configs). I'll have to recheck that - at one point I did need to add an additional name to the certificate to get things to match between the inside and outside kubernetes views of the world.

And then note that when launching the service in the IDE (as a SpringBoot app), the profiles that are enabled are: local,default,native,kubernetes,ide

As I've stated a few times, I might be able to simplify that list a little but haven't tried.  It grew a bit as I was trying to work with GraalVM and was copying settings back and forth between projects at one point.  

Hoping your day is blessed!

Scott

 

 

 

 

 

 

Local Development, Microk8s Built-in registry and DNS Parity - Part 2

In part 1, I setup the local Microk8s registry but treat it like a private registry. I was also able to build and deploy a containerized image to Microk8s.  The resulting application doesn't work yet though and in the POD logs it indicated a problem with configuration.

The error is:

Can't read configMap with name: [starcases-sb-note] in namespace:[default]. Ignoring

In this particular case, that is a bit misleading.  I have experienced a number of issues with similar results.  The root of those issues tended to be one of:

• Firewall rules - there are some complexities related to IPTables vs NFTables and a few other items that can interact.
• Configuration mistakes or incomplete configuration for the kubernetes client portion. This commonly occurs when you specify IP addresses for resources that can/do change and you don't keep configuration in sync - usually this is something like the DB server.
• Overlapping / conflicting configuration between spring configuration files - application.properties and bootstrap.yaml, etc.

Or in this case, it was application issue which presents itself early in the process.  I had started to add some new data to the service and didn't fully implement the entity and repository classes which resulted in causing the failure during the initial configuration.

The easiest way to debug that was to start the application locally in Eclipse - at which point the console logs produced in Eclipse were clear about what was missing. I did try to do some mild debugging in the Microk8s cluster but the results were not clear and in fact had me looking at different issues.

So at this point, the cluster is up and the database and service are running.  Here is snapshot of the system as a whole.

A quick interlude to describe some of data I'm working with and some process aspects.  This "pet" project started with a simply greeting service.  The goal I had for that was simply to work with multiple languages/locales. Using UTF-8, store variations of "Hi" from multiple languages in a DB table and associate each with a locale. I worked in multi-language environments a bit and always found it interesting. As Microservices and other ideas come up, I've wondered about alternative methods of handling things.  I got distracted one day with that thought and found a dataset of country names where all countries were represented in a number of different languages.  This happened to be a XLIFF 1.2 document.  I ended up writing some code that allowed me to use it a bit like resource bundle - give a key and locale as parameters and get back the data in that target locale. Nothing fancy, but interesting.  The challenge in my mind would be how to use it in a way that minimizes the chance of invalid keys showing up in "client code" over time.

Back to the database for a minute. Here is a quick image of that tables.

Note that databasechangelog and databasechangeloglock are related to Liquibase.  The greet table is what I mentioned before.  The notes and users tables are simply notes and user info - I'm in the process of integrating something like OAuth into the system so the users table could go away (the data would likely migrate to some other store).  I'm still trying to evaluate a few packages in this area but that is a bit slow right now.

The spatial_ref_sys table exist because I went back and wanted to implement this on top of Postgis to potentially leverage the geographic support.  

The globalterrorism table is a kaggle dataset that I imported into Postgres. I'm not sure whether I should have been surprised at the mild difficulties in getting that data imported.  I did a bit of quick analysis on the data to determine types and such but even so I ran into some issues.  I ended up installing a utility package which provided a csv stat feature that dumped information about each column after analyzing all rows and columns.  This helped provide correct info to fix a number of items where columns had few values.  Some of the data has latitude / longitude which fit in with my thought of Postgis.

Additionally, I started to toy with something called "Geoserver" which is used for working with mapping / spatial data. I'd like to tie all the items that I am working with together in some way. Maybe integrate a few more data sets related to food, education, crime, cost of living, terrorism and do some sort of "heat map" or other representation for areas with "fewer issues" and/or "more benefits". Anyways, it is just a learning experiment overall.

Ok, back from the interlude.  Some notes on the service implementation side.  

While I have the service running in Microk8s; I am also able to run it in Eclipse.

And postman calls confirm that the services function.

I probably should have returned a 204 response since I didn't return a body but for a quick test this was ok.

The controller looks like the following.

And an entity looking like this:

In parallel to this, I'm also implementing the same basic data/logic as a Quarkus application.  That will be a post for another day.

So overall, things function.  I with for better DNS integration which I will be trying to improve either through customization to CoreDNS config in Microk8s or via running Unbound locally.  That is still in the research phase.

I'm starting a part 3 post with some additional details and notes. As I am going back through my code and setup, I am finding a few things to cleanup and a few things to change and document. I'll go into code a bit more and also configuration which is probably the hardest part to get done "well".

Have a blessed day!

Scott

[Update 2021/03] I'm finding that a number of things break and need minor changes as I update to the most recent releases of items in this tech stack. The biggest items in the overall stack are:

• Microk8s v1.20.2
• SpringBoot 2.4.3
• SpringCloud 2020.0.1
• Eclipse JKube 1.1.1
• Java 11
• GraalVM 21.0.0
• Liquibase Maven plugin 4.2.0
• Eclipse 20-12 (4.18.0)
• Ubuntu 20.10

A few "random" issues after updates for Microk8s, SpringBoot, SpringCloud, JKube has required some minor tweaks to settings.  I'm still working out improvements for a few settings.

Note that Microk8s is installed via Snap.  Snap is ok but I prefer a bit more control over some packages. I stopped using the snap installed version of Eclipse because of some issues I had run into a while back - seems improved by a local install of Eclipse.  Note that Ubuntu 20.10 isn't an LTS version and that also causes some headaches.  I've had to fight some overall system issues off and on because of my dual graphics card setup (on-chip Intel UHD 630 and a discrete AMD Radeon card). I think this is working ok now but I'm a little disappointed in the setup - I may document that in a different post.

I finally broke down and moved my home directory to an nvme based drive (from standard hard disk) to speed up some activities. That was actually a bit of a painful move - I didn't remember adding an ACL to my home directory. I'd like to move other a few other items to an SSD but I am taking more care because I read some articles indicating that there might be some dependencies across some of the directories/items I'd like to move. The move to the NVME drive drastically helps Eclipse which I  installed off my home directory.

At the service level of my simple SpringBoot based service, I am removing the paging support I started to implement for now. I am still trying to find time to work on this overall and want to have an equivalent Quarkus based service for comparison. I ended up reworking some of the annotations related to using UUID for primary keys since Eclipse started flagging some errors after something changed recently.  I believe that issue is resolved for now. 

A few quick notes on performance. I tend to watch 'top' quite a bit to see what is using resources and also keep and eye on logs in /var/log/.  One issue I fought for a while was logging.  In some cases, some of the services for Microk8s end up producing substantial log out.  In some cases, it is all trace/debug/info. I updated Journald to only log at the warning level - this brings the average system load down nicely.  I also ran across issues with Gnome causing excessive logging (about errors).  I found a reported issue with a patch which fixed it and reduced the logging and overall system load a bit. At this point, kube-apiserver tends to run above 10% CPU on average and I'd like to understand why and whether there is a way to reduce that. I'm also watching for "parasitic" losses from some items I don't need/want.  After the Ubuntu 20.10 upgrade, I found a few services that load but either don't run or they produce needless error logs.  "Masking" those services so they won't/can't start has helped with the parasitic losses.

That's all the time I have for update now.  My regular work and the multitude of home projects are eating up most time now.

Local Development, Microk8s Built-in registry and DNS Parity - Part 1

When I do development at home, I prefer processes that feel "Production Like" in some sense.  For instance, a preference for DNS names in place of IP addresses in configuration data. 

I've been working with Microk8s for a while now at home and I decided to use the local registry it offers as an option.  Why? Well, I want to mimic remote registry functionality and also enable working with some other non-docker container solutions.  I'm checking out different container technologies and the built-in registry uses a newer Containerd version than available in the public install of Docker-CE. I want to analyze the benefits promoted by the newer version a bit. I'm also looking at how similar/seamless a IDE based local deploy of a SpringBoot based service can be compared to deploying into Microk8s.

This post mentions some general ideas I experimented with - there are other solutions that are possibly better and easier. This path helped me learn a few things I might not have otherwise though.

The Microk8s built-in registry documentation expects you to use a convention along the lines of:

    localhost:32000/<appname>:registry

which works fine... but I want to mimic remote registry behavior a bit closer.  I really prefer a scheme like one of the following: 

• docker-star-cases.starcases.com:32000/starcases/<appname>:<version>
• docker-star-cases/starcases/<appname>:<version>

where "star cases" is a fun way I represent my family using the initials of our names.  These mimic the general convention <registry>/<user>/<appname>:<version> where <user> is often an individual user, organization name,  team name, etc.

There are some questions that this goal creates directly related to the registry itself.

1. How do you get the image into the Microk8s built-in registry?
2. How are image references outside of Microk8s resolved?
3. How are image references in deployment descriptors resolved inside Microk8s?

Those questions tend to equate to "How do I resolve a name to a resource identifier (IP or DNS name) from different contexts?".  The 2 initial contexts are:

• Local host (workstation)
• Inside a Microk8s node

There are at least a few ways to do this.  Fundamentally, I want to alias the built-in registry with a name I chose but I want to avoid modifying the existing service as much as possible.

Note that K8s service DNS aliases came up at one time but lack of agreement seemingly killed the idea.

https://github.com/kubernetes/kubernetes/issues/39792

Some general methods available include:

• Editing workstation /etc/hosts to map 'docker-star-cases' to an appropriate IP
• At the home router level, create a static DNS record for 'docker-star-cases' with an appropriate IP
• Possibly using the External DNS module to expose the names outside of Microk8s
• Expose CoreDNS somehow and implement some sort of split horizon

For now, I picked a bit of a compromise between functionality and complexity.  The simplest method would involve updating the /etc/hosts file to provide DNS name to IP mappings for use outside of the Microk8s cluster.  I decided that was too limiting and wanted to stretch the idea a bit further. I really wanted integration with my home DNS.

I'll admit that I did take the easy road here for now.  In my home router, I'm able to add user defined DNS mappings. In this case, I added DNS names of "docker-star-cases" and "docker-star-cases.star-cases.com" with a private IP of 192.x.x.x.. Now, from a terminal window; "nslookup docker-star-cases" returns that private IP address. Accessing the registry didn't work immediately because the registry is setup as insecure and my docker tooling needs configuration to tell it that the insecure registry is ok to use.

The Microk8s instructions mention this but they use their default naming proposal.  So instead of "localhost", I edited "/etc/docker/daemon.json" and added:

{

            "insecure-registries" : ["docker-star-cases:32000"]

         }

and restarted the docker daemon as the Microk8s instructions indicate:

   sudo systemctl restart docker

So at this point; I can generate and tag images with something that resolves to my system using the naming I desired.  The result looks like:

docker-star-cases.starcases.com:32000/starcases/sb-note:latest

and in my local Docker cache (not Microk8s registry) I find:

docker-star-cases.starcases.com:32000/starcases/sb-note   latest                  7a9e91bdf139   41 years ago    311MB

I'm not advocating using 'latest' here but it works well for the example. Note that the "41 years" is related to using JKube.  

The program I am working on is a service implemented by a SpringBoot app which uses (the now) org.eclipse.jkube:kubernetes-maven-plugin to support generating the required kubernetes artifacts and support deployment. The services are just various ideas I'm testing out in general. The service will use a PostgreSQL database.

See 'Prep work' at end of this post for more info on the DB setup.

For now, I'm just using the default namespace. It will probably make sense to utilize other namespaces as needs grow. 

Note that I have a service setup for the DB - the intent is to use that DB for local dev/testing using my IDE and also by the same service deployed within the k8s cluster. This will come up again later.

NAME                                       READY   STATUS    RESTARTS   AGE

pod/springboot-note-db1-57dd9588c4-6x822   1/1     Running   0          2m34s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE

service/kubernetes                ClusterIP   10.152.183.1    <none>        443/TCP    77m

service/springboot-note-db1-svc   ClusterIP   10.152.183.50   <none>        5944/TCP   46m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/springboot-note-db1   1/1     1            1           2m35s

NAME                                             DESIRED   CURRENT   READY   AGE

replicaset.apps/springboot-note-db1-57dd9588c4   1         1         1       2m34s

Using Eclipse and Jkube, I can create and deploy my service now - but that doesn't mean it runs yet. 

[INFO] --- kubernetes-maven-plugin:1.1.0:apply (default-cli) @ starcases-sb-note ---

[INFO] k8s: Using Kubernetes at https://kubernetes.default.svc:16443/ in namespace default with manifest /home/scott/src/eclipse-workspace/SpringBootNote/target/classes/META-INF/jkube/kubernetes.yml 

[INFO] k8s: Updating Service from kubernetes.yml

[INFO] k8s: Updated Service: target/jkube/applyJson/default/service-springboot-note-db1-svc.json

[INFO] k8s: Creating a Service from kubernetes.yml namespace default name starcases-sb-note

[INFO] k8s: Created Service: target/jkube/applyJson/default/service-starcases-sb-note.json

[INFO] k8s: Updating ConfigMap from kubernetes.yml

[INFO] k8s: Updated ConfigMap: target/jkube/applyJson/default/configmap-springboot-note-db1.json

[INFO] k8s: Updating ConfigMap from kubernetes.yml

[INFO] k8s: Updated ConfigMap: target/jkube/applyJson/default/configmap-starcases-sb-note.json

[INFO] k8s: Creating a Deployment from kubernetes.yml namespace default name starcases-sb-note

[INFO] k8s: Created Deployment: target/jkube/applyJson/default/deployment-starcases-sb-note.json

[INFO] k8s: HINT: Use the command `kubectl get pods -w` to watch your pods start up

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

Outside of Microk8s; I can see the image in the registry

$  curl http://docker-star-cases.starcases.com:32000/v2/_catalog

{"repositories":["starcases/sb-note"]}

Here is the microk8s listing after the build/deploy from Eclipse for the service.

NAME                                       READY   STATUS             RESTARTS   AGE

pod/springboot-note-db1-57dd9588c4-6x822   1/1     Running            0          75m

pod/sb-note-8874fcfb-lltw5                 0/1     ImagePullBackOff   0          57m

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

service/kubernetes                ClusterIP   10.152.183.1     <none>        443/TCP    150m

service/springboot-note-db1-svc   ClusterIP   10.152.183.50    <none>        5944/TCP   119m

service/sb-note                   ClusterIP   10.152.183.102   <none>        8080/TCP   57m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/springboot-note-db1   1/1     1            1           75m

deployment.apps/sb-note               0/1     1            0           57m

NAME                                             DESIRED   CURRENT   READY   AGE

replicaset.apps/springboot-note-db1-57dd9588c4   1         1         1       75m

replicaset.apps/sb-note-8874fcfb                 1         1         0       57m

It deployed but doesn't run - the POD logs show that it failed to pull the image.

$ kubectl logs starcases-sb-note-8874fcfb-lltw5

Error from server (BadRequest): container "spring-boot" in pod "sb-note-8874fcfb-lltw5" is waiting to start: trying and failing to pull image

Since we are really trying to treat this as a private registry, if you review the Microk8s documentation for private registries you'll find some setup is needed to allow this new named system to act as a registry mirror. An addition is needed in the file:

    /var/snap/microk8s/current/args/containerd-template.toml

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker-star-cases.starcases.com:32000"]

        endpoint = ["http://docker-star-cases.starcases.com:32000"]

At this point, you need to perform a stop/start on the Microk8s cluster.

$ microk8s stop; microk8s start

And if you check pods now you will find that it did start.

    default              pod/starcases-sb-note-8874fcfb-6g2sr          1/1     Running   1          75s

If you check the POD logs you will find a problem though.  The log entry of interest is:

2021-02-18 06:10:04.108  WARN 1 --- [           main] s.c.k.f.c.Fabric8ConfigMapPropertySource : Can't read configMap with name: [starcases-sb-note] in namespace:[default]. Ignoring.

What?  Did I mention that I want the majority of the service configuration to reside in a config map (and/or secret later) regardless of whether the service is running in the Microk8s cluster or from my Eclipse IDE?  This allows me to work with just one primary source of configuration and allows for some overrides if needed.  We'll, the issue with the service startup is that it was unable to find the configmap and therefore the database connection info was unavailable.

I'll continue this in a Part 2 post to demonstrate a method to get it working and some other details.

[Edit 2021/02/15] 

• NOTES
• I had issues with 'microk8s status' telling me that it wasn't running even though it would show the services up.  This may be due to me creating a soft link to  /snap/bin/microk8s.kubectl using the name kubectl.  I also had an alias of 'kubectl=microk8s kubectl'. 
• Prep Work
• Initial Microk8s initial install / config.
• sudo snap install microk8s --classic

• microk8s enable dns 
• microk8s enable registry storage
• mkdir ~/.kube
• NOTE: Backup any existing ~./.kube/config if needed!
• microk8s config > ~/.kube/config
• Database config map and deployment yamls
• sprngboot-note-db1-configmap.yaml

• apiVersion: v1
  kind: ConfigMap
  metadata:
    name: springboot-note-db1
  data:
    POSTGRES_PASSWORD: PASSWORD 

• springboot-note-db1-deployment.yaml

• kind: Service

  apiVersion: v1

  metadata:

    name:  springboot-note-db1-svc

  spec:

    ports:

      - port: 5944

        targetPort: 5432

        name:  springboot-note-db1-svc

        protocol: TCP

    selector:

      app: springboot-note-db1

  ---

  kind: Deployment

  apiVersion: apps/v1

  metadata:

    annotations:

      configmap.jkube.io/update-on-change: springboot-note-db1

    name: springboot-note-db1

  spec:

    replicas: 1

    selector:

       matchLabels:

          app: springboot-note-db1

    template:

      metadata:

          labels:

            app: springboot-note-db1

      spec:

          containers:

            - name: springboot-note-db1

              image: postgis/postgis:13-3.1

              ports:

                - containerPort: 5432

              envFrom:

                - configMapRef:

                   name: springboot-note-db1

• Applying the settings 
• kubectl apply -f sprngboot-note-db1-configmap.yaml
• kubectl apply -f springboot-note-db1-deployment.yaml

[edit 2021/04/04] Note that wanting to resolve a 1 part name (docker-star-cases) to docker-star-cases.star-cases.com (and then to the final IP address) likely requires setting the search domain.  i.e. on Ubunto 20.10, I: 

• sudo vi /etc/systemd/resolved.conf

and I edit it so under the [Resolve] section, there is a line of:

Domains=star-cases.com