Local Development, Microk8s Built-in registry and DNS Parity - Part 2

In part 1, I setup the local Microk8s registry but treat it like a private registry. I was also able to build and deploy a containerized image to Microk8s.  The resulting application doesn't work yet though and in the POD logs it indicated a problem with configuration.

The error is:

Can't read configMap with name: [starcases-sb-note] in namespace:[default]. Ignoring

In this particular case, that is a bit misleading.  I have experienced a number of issues with similar results.  The root of those issues tended to be one of:

• Firewall rules - there are some complexities related to IPTables vs NFTables and a few other items that can interact.
• Configuration mistakes or incomplete configuration for the kubernetes client portion. This commonly occurs when you specify IP addresses for resources that can/do change and you don't keep configuration in sync - usually this is something like the DB server.
• Overlapping / conflicting configuration between spring configuration files - application.properties and bootstrap.yaml, etc.

Or in this case, it was application issue which presents itself early in the process.  I had started to add some new data to the service and didn't fully implement the entity and repository classes which resulted in causing the failure during the initial configuration.

The easiest way to debug that was to start the application locally in Eclipse - at which point the console logs produced in Eclipse were clear about what was missing. I did try to do some mild debugging in the Microk8s cluster but the results were not clear and in fact had me looking at different issues.

So at this point, the cluster is up and the database and service are running.  Here is snapshot of the system as a whole.

A quick interlude to describe some of data I'm working with and some process aspects.  This "pet" project started with a simply greeting service.  The goal I had for that was simply to work with multiple languages/locales. Using UTF-8, store variations of "Hi" from multiple languages in a DB table and associate each with a locale. I worked in multi-language environments a bit and always found it interesting. As Microservices and other ideas come up, I've wondered about alternative methods of handling things.  I got distracted one day with that thought and found a dataset of country names where all countries were represented in a number of different languages.  This happened to be a XLIFF 1.2 document.  I ended up writing some code that allowed me to use it a bit like resource bundle - give a key and locale as parameters and get back the data in that target locale. Nothing fancy, but interesting.  The challenge in my mind would be how to use it in a way that minimizes the chance of invalid keys showing up in "client code" over time.

Back to the database for a minute. Here is a quick image of that tables.

Note that databasechangelog and databasechangeloglock are related to Liquibase.  The greet table is what I mentioned before.  The notes and users tables are simply notes and user info - I'm in the process of integrating something like OAuth into the system so the users table could go away (the data would likely migrate to some other store).  I'm still trying to evaluate a few packages in this area but that is a bit slow right now.

The spatial_ref_sys table exist because I went back and wanted to implement this on top of Postgis to potentially leverage the geographic support.  

The globalterrorism table is a kaggle dataset that I imported into Postgres. I'm not sure whether I should have been surprised at the mild difficulties in getting that data imported.  I did a bit of quick analysis on the data to determine types and such but even so I ran into some issues.  I ended up installing a utility package which provided a csv stat feature that dumped information about each column after analyzing all rows and columns.  This helped provide correct info to fix a number of items where columns had few values.  Some of the data has latitude / longitude which fit in with my thought of Postgis.

Additionally, I started to toy with something called "Geoserver" which is used for working with mapping / spatial data. I'd like to tie all the items that I am working with together in some way. Maybe integrate a few more data sets related to food, education, crime, cost of living, terrorism and do some sort of "heat map" or other representation for areas with "fewer issues" and/or "more benefits". Anyways, it is just a learning experiment overall.

Ok, back from the interlude.  Some notes on the service implementation side.  

While I have the service running in Microk8s; I am also able to run it in Eclipse.

And postman calls confirm that the services function.

I probably should have returned a 204 response since I didn't return a body but for a quick test this was ok.

The controller looks like the following.

And an entity looking like this:

In parallel to this, I'm also implementing the same basic data/logic as a Quarkus application.  That will be a post for another day.

So overall, things function.  I with for better DNS integration which I will be trying to improve either through customization to CoreDNS config in Microk8s or via running Unbound locally.  That is still in the research phase.

I'm starting a part 3 post with some additional details and notes. As I am going back through my code and setup, I am finding a few things to cleanup and a few things to change and document. I'll go into code a bit more and also configuration which is probably the hardest part to get done "well".

Have a blessed day!

Scott

[Update 2021/03] I'm finding that a number of things break and need minor changes as I update to the most recent releases of items in this tech stack. The biggest items in the overall stack are:

• Microk8s v1.20.2
• SpringBoot 2.4.3
• SpringCloud 2020.0.1
• Eclipse JKube 1.1.1
• Java 11
• GraalVM 21.0.0
• Liquibase Maven plugin 4.2.0
• Eclipse 20-12 (4.18.0)
• Ubuntu 20.10

A few "random" issues after updates for Microk8s, SpringBoot, SpringCloud, JKube has required some minor tweaks to settings.  I'm still working out improvements for a few settings.

Note that Microk8s is installed via Snap.  Snap is ok but I prefer a bit more control over some packages. I stopped using the snap installed version of Eclipse because of some issues I had run into a while back - seems improved by a local install of Eclipse.  Note that Ubuntu 20.10 isn't an LTS version and that also causes some headaches.  I've had to fight some overall system issues off and on because of my dual graphics card setup (on-chip Intel UHD 630 and a discrete AMD Radeon card). I think this is working ok now but I'm a little disappointed in the setup - I may document that in a different post.

I finally broke down and moved my home directory to an nvme based drive (from standard hard disk) to speed up some activities. That was actually a bit of a painful move - I didn't remember adding an ACL to my home directory. I'd like to move other a few other items to an SSD but I am taking more care because I read some articles indicating that there might be some dependencies across some of the directories/items I'd like to move. The move to the NVME drive drastically helps Eclipse which I  installed off my home directory.

At the service level of my simple SpringBoot based service, I am removing the paging support I started to implement for now. I am still trying to find time to work on this overall and want to have an equivalent Quarkus based service for comparison. I ended up reworking some of the annotations related to using UUID for primary keys since Eclipse started flagging some errors after something changed recently.  I believe that issue is resolved for now. 

A few quick notes on performance. I tend to watch 'top' quite a bit to see what is using resources and also keep and eye on logs in /var/log/.  One issue I fought for a while was logging.  In some cases, some of the services for Microk8s end up producing substantial log out.  In some cases, it is all trace/debug/info. I updated Journald to only log at the warning level - this brings the average system load down nicely.  I also ran across issues with Gnome causing excessive logging (about errors).  I found a reported issue with a patch which fixed it and reduced the logging and overall system load a bit. At this point, kube-apiserver tends to run above 10% CPU on average and I'd like to understand why and whether there is a way to reduce that. I'm also watching for "parasitic" losses from some items I don't need/want.  After the Ubuntu 20.10 upgrade, I found a few services that load but either don't run or they produce needless error logs.  "Masking" those services so they won't/can't start has helped with the parasitic losses.

That's all the time I have for update now.  My regular work and the multitude of home projects are eating up most time now.