SpringBoot Batch load CSV into Postgresql - external with K8s configmap

 This is a short demo of running a SpringBoot batch job in eclipse where the configuration is retrieved from a Microk8s configmap.  The setup can be extended to use K8s secrets but wasn't done here yet. 

I had worked with several other tools when initially trying to import this data into my database but I ran into a number of problems with the methods.  The biggest issue is some "complex" data as part of a set of fields that most import tools/processes don't handle correctly but where handled here fine.  It did take a few iterations to work out a few incorrect field data types but Spring Batch provides pretty clear information regarding the problematic fields and types.

 

 

Here you can see an example of what wasn't handled well by some other import methods. It usually included partially quoted data and also commas.

 Here is a quick screen shot of the Microk8s info showing my Postgresql database. My batch job uses the service name as the host portion of the connection string. For now, I manually update my DNS resolutions to map the service name to the service IP from outside of my cluster. I'm working toward that being an integrated/automated item requiring no manual intervention.

Thanks for looking and have a blessed day!

Scott

 

Local Development, Microk8s Built-in registry and DNS Parity - Part 3

In part 2 of this, I had a SpringBoot service which was runnable from both an IDE and from Microk8s - with the configuration for both methods coming from a configmap in Microkk8s.

Here I want to go into the configuration a bit more because that part created a few headaches but I think it is sorted out now (for the most part).

To start with, one of the major enablers for this setup is a bootstrap.yml file.  In this case, I'm not using any application.properties file in the apps main/resources folder.  The use of the bootstrap file is to get required settings, well - bootstrapped before the majority of Spring auto-configuration starts up.

 Spring profiles are also a key part of this setup. The main spring config in the bootstrap file is defined for the profiles:

• native
• default
• kubernetes

 I'll admit that I should have documented this better initially - a key aspect is the kubernetes profile though. There are likely other ways to get things working but so far this is cleanest and most reliable. Note that the kubernetes profile is used even when executing the service in the IDE.  I'll explain that a bit more in minute but first lets look at the initial part of the bootstrap configuration.

 spring:
  config:
    activate:
      on-profile:
      - native
      - default
      - kubernetes
  application:
      name: sb-note
  cloud:
    discovery:
      enabled: true
    kubernetes:
      enabled: true    
      config:
        enabled: true
        enable-api: true
        paths:
        - /deployments/config/application.properties
      discovery:
        enabled: true
      client:
        namespace: default
        clientCertFile: /var/snap/microk8s/current/certs/kubelet.crt
        caCertFile: /var/snap/microk8s/current/certs/kubelet.crt
        ca-cert-file: /var/snap/microk8s/current/certs/kubelet.crt
        trust-certs: true
        masterUrl: https://kubernetes:443
springdoc:
  swagger-ui.path: /swagger-ui.html
debug: true

 

Note that the various cloud/kubernetes aspects are enabled.  These are important for the dependencies:

• org.springframework.cloud:spring-cloud-starter-bootstrap
  
• org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8-config
•  org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8
•  org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8-all

I may be able to simplify those a bit but that is what I have after thrashing a bit with a recent SpringCloud update. The combination provides the ability to "know about" kubernetes and use features like configmaps.  Since I always want to pull my main configuration using a configmap, I always use the kubernetes profile and the enablements above so configmaps are usable.  

Note that the "client" section is what generally provides any configuration needed so your app can leverage the "kubernetes" awareness and access those associated resources. Note that the client.masterUrl though is defined in the manner which the service needs it when it actually is running in Microk8s.  The https://kubernetes:443 is not accessible from outside of Microk8s.  So this config is all I needed for running in Kubernetes - but what about when running it from outside?

Well, there is a second part to the above bootstrap.yml file which is:

---
spring:
  config:
    activate:
      on-profile:
      - ide
  cloud:
    kubernetes:
      client:
        masterUrl: https://kubernetes.default.svc:16443

and when I run the service in the IDE, I add an additional spring profile to end of my list of active profiles: ide

The result is that the client.masterUrl of https://kubernetes.default.svc:16443 overrides the previous definition due to ordering. The kubernetes.default.svc is what I have been manually maintaining an IP for in my home routers DNS.  That DNS name resolves to the cluster IP of the kubernetes cluster and the port 16443 which is what Microk8s exposes it on - see their documentation on ports. 

So this allows the service to see and access kubernetes and the associated resources from both inside and outside of the cluster. Note that I may be able to move some of the other client settings to the "ide" profile but haven't gotten quite that far in my cleanup. I believe that this can be done because I don't think the various paths for the client section are probably visible as shows show - so are probably ignored when running inside kubernetes. 

Note, initially I had things working in a somewhat awkward fashion which this setup fixes. If you don't override the masterUrl then you can still make things work but it is odd.  Basically, if your deployment exposes the configmap as both a configmap AND an application.properties file - when running in kubernetes and the masterUrl doesn't resolve correctly, the configmap won't be found but the application.properties file is mounted and loaded via normal spring auto-configuration.  You'll see some error/warnings in logs but configuration is available.  Once I realized that is what I had done, I worked out this additional profile so to prevent that configuration from occurring.   The current config shown above doesn't result in those error conditions.  

 Note that the configmap is defined as the following.

namespace: default
data:
  application.properties: |-
    spring.profiles.active=kubernetes
    debug=false
    spring.application=sb-note
    spring.datasource.url=jdbc:postgresql://springboot-note-db1-svc.default.svc:5944/postgres
    spring.datasource.username=postgres
    spring.datasource.password=PASSWORD
    spring.jpa.database=POSTGRESQL
    spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
    spring.jpa.hibernate.ddl-auto=update
    springdoc.swagger-ui.path=/swagger-ui.html
    logging.level.org.hibernate.SQL=DEBUG
    logging.level.org.hibernate.type=WARN
    spring.jpa.show-sql=true
    spring.jpa.properties.hibernate.format_sql=true
    javax.persistence.schema-generation.database.action=update

So the configmap data must be available to the service in some form or the database connection initialization will fail.  That is the one of the most visible ways you will know that it isn't finding the information - which is true while running in the IDE and in k8s as well.   I'll note that I am not using k8s secrets here yet - that would be the next step in doing a more "enterprise" setup. That should work in a similar fashion to the configmap setup.

 Note that so far, all this setup is for actual runtime and not deployment - although the above configmap is defined in the way that Jkube needs it so as to create the actual configmap during deployment.  

 For build/deployment, a few key properties are needed in the maven pom file.

        <java.version>11</java.version>
        <docker.starcases.registry>docker-star-cases.starcases.com:32000</docker.starcases.registry>
        <image.user>starcases</image.user>
        <spring-cloud.version>2020.0.1</spring-cloud.version>

        <jkube.masterUrl>https://kubernetes.default.svc:16443</jkube.masterUrl>
        <jkube.trustStoreFile>/var/snap/microk8s/current/certs/kubelet.crt</jkube.trustStoreFile>
        <jkube.version>1.1.1</jkube.version>
        <jkube.recreate>true</jkube.recreate>
        <jkube.deploy.ignoreServices>true</jkube.deploy.ignoreServices>
        <jkube.generator.name>${docker.starcases.registry}/${image.user}/${project.artifactId}:${project.version}</jkube.generator.name>
        <jkube-revision-history>1</jkube-revision-history>

 Note that I am having springboot build the service and JKube is repackaging into a docker compatible image and then deploying that image to the Microk8s registry and then deploying the other resources Jkube generates configuration for from my setup.

The main build is being done using the recent SpringBoot 2.3 feature enhancements - the pom.xml part of interest is.

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <excludeDevtools>false</excludeDevtools>
                    <release>11</release>
                    <image>
                        <name>${jkube.generator.name}</name>
                    </image>
                    <layers>
                        <enabled>true</enabled>
                        <includeLayerTools>true</includeLayerTools>
                    </layers>
                </configuration>
            </plugin>
        </plugins>

 And then rest of the repackaging and deployment is handled by:

        <profile>
            <id>kubernetes</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.eclipse.jkube</groupId>
                        <artifactId>kubernetes-maven-plugin</artifactId>
                        <version>${jkube.version}</version>

                        <configuration>
                            <profile>minimal</profile>
                            <buildStrategy>docker</buildStrategy>
                            <detail>true</detail>
                            <imagePullPolicy>Always</imagePullPolicy>

                            <logStdout>true</logStdout>
                            <verbose>true</verbose>

                            <failOnNoKubernetesJson>true</failOnNoKubernetesJson>
                            <failOnValidationError>true</failOnValidationError>

                            <skipExtendedAuth>true</skipExtendedAuth>
                            <pushRegistry>${docker.starcases.registry}</pushRegistry>

                            <skipBuild>true</skipBuild>
                            <kubernetesManifest>${basedir}/target/classes/META-INF/jkube/kubernetes.yml</kubernetesManifest>

                            <resources>
                                <labels>
                                    <all>
                                        <property>
                                            <name>app</name>
                                            <value>${project.artifactId}</value>
                                        </property>

                                        <property>
                                            <name>provider</name>
                                            <value>jkube</value>
                                        </property>
                                        <property>
                                            <name>group</name>
                                            <value>${project.groupId}</value>
                                        </property>
                                        <property>
                                            <name>version</name>
                                            <value>${project.version}</value>
                                        </property>

                                    </all>
                                </labels>
                            </resources>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>

There are likely a few items I can cleanup here but it is functional.  

 Note that the maven goal I use in the IDE for the Microk8s build / deployment is:

• clean  spring-boot:build-image  k8s:resource  k8s:push  k8s:apply

And I specify a profile of "kubernetes".  I do have a few environmental settings as well - maven.test.skip, skip tests, and an item for trusting the kubernetes certificate which I may not need now (at one point I worked out some issues related to it but am noticing I didn't clean all settings in all the IDE launch configs). I'll have to recheck that - at one point I did need to add an additional name to the certificate to get things to match between the inside and outside kubernetes views of the world.

And then note that when launching the service in the IDE (as a SpringBoot app), the profiles that are enabled are: local,default,native,kubernetes,ide

As I've stated a few times, I might be able to simplify that list a little but haven't tried.  It grew a bit as I was trying to work with GraalVM and was copying settings back and forth between projects at one point.  

Hoping your day is blessed!

Scott

 

 

 

 

 

 

Local Development, Microk8s Built-in registry and DNS Parity - Part 2

In part 1, I setup the local Microk8s registry but treat it like a private registry. I was also able to build and deploy a containerized image to Microk8s.  The resulting application doesn't work yet though and in the POD logs it indicated a problem with configuration.

The error is:

Can't read configMap with name: [starcases-sb-note] in namespace:[default]. Ignoring

In this particular case, that is a bit misleading.  I have experienced a number of issues with similar results.  The root of those issues tended to be one of:

• Firewall rules - there are some complexities related to IPTables vs NFTables and a few other items that can interact.
• Configuration mistakes or incomplete configuration for the kubernetes client portion. This commonly occurs when you specify IP addresses for resources that can/do change and you don't keep configuration in sync - usually this is something like the DB server.
• Overlapping / conflicting configuration between spring configuration files - application.properties and bootstrap.yaml, etc.

Or in this case, it was application issue which presents itself early in the process.  I had started to add some new data to the service and didn't fully implement the entity and repository classes which resulted in causing the failure during the initial configuration.

The easiest way to debug that was to start the application locally in Eclipse - at which point the console logs produced in Eclipse were clear about what was missing. I did try to do some mild debugging in the Microk8s cluster but the results were not clear and in fact had me looking at different issues.

So at this point, the cluster is up and the database and service are running.  Here is snapshot of the system as a whole.

A quick interlude to describe some of data I'm working with and some process aspects.  This "pet" project started with a simply greeting service.  The goal I had for that was simply to work with multiple languages/locales. Using UTF-8, store variations of "Hi" from multiple languages in a DB table and associate each with a locale. I worked in multi-language environments a bit and always found it interesting. As Microservices and other ideas come up, I've wondered about alternative methods of handling things.  I got distracted one day with that thought and found a dataset of country names where all countries were represented in a number of different languages.  This happened to be a XLIFF 1.2 document.  I ended up writing some code that allowed me to use it a bit like resource bundle - give a key and locale as parameters and get back the data in that target locale. Nothing fancy, but interesting.  The challenge in my mind would be how to use it in a way that minimizes the chance of invalid keys showing up in "client code" over time.

Back to the database for a minute. Here is a quick image of that tables.

Note that databasechangelog and databasechangeloglock are related to Liquibase.  The greet table is what I mentioned before.  The notes and users tables are simply notes and user info - I'm in the process of integrating something like OAuth into the system so the users table could go away (the data would likely migrate to some other store).  I'm still trying to evaluate a few packages in this area but that is a bit slow right now.

The spatial_ref_sys table exist because I went back and wanted to implement this on top of Postgis to potentially leverage the geographic support.  

The globalterrorism table is a kaggle dataset that I imported into Postgres. I'm not sure whether I should have been surprised at the mild difficulties in getting that data imported.  I did a bit of quick analysis on the data to determine types and such but even so I ran into some issues.  I ended up installing a utility package which provided a csv stat feature that dumped information about each column after analyzing all rows and columns.  This helped provide correct info to fix a number of items where columns had few values.  Some of the data has latitude / longitude which fit in with my thought of Postgis.

Additionally, I started to toy with something called "Geoserver" which is used for working with mapping / spatial data. I'd like to tie all the items that I am working with together in some way. Maybe integrate a few more data sets related to food, education, crime, cost of living, terrorism and do some sort of "heat map" or other representation for areas with "fewer issues" and/or "more benefits". Anyways, it is just a learning experiment overall.

Ok, back from the interlude.  Some notes on the service implementation side.  

While I have the service running in Microk8s; I am also able to run it in Eclipse.

And postman calls confirm that the services function.

I probably should have returned a 204 response since I didn't return a body but for a quick test this was ok.

The controller looks like the following.

And an entity looking like this:

In parallel to this, I'm also implementing the same basic data/logic as a Quarkus application.  That will be a post for another day.

So overall, things function.  I with for better DNS integration which I will be trying to improve either through customization to CoreDNS config in Microk8s or via running Unbound locally.  That is still in the research phase.

I'm starting a part 3 post with some additional details and notes. As I am going back through my code and setup, I am finding a few things to cleanup and a few things to change and document. I'll go into code a bit more and also configuration which is probably the hardest part to get done "well".

Have a blessed day!

Scott

[Update 2021/03] I'm finding that a number of things break and need minor changes as I update to the most recent releases of items in this tech stack. The biggest items in the overall stack are:

• Microk8s v1.20.2
• SpringBoot 2.4.3
• SpringCloud 2020.0.1
• Eclipse JKube 1.1.1
• Java 11
• GraalVM 21.0.0
• Liquibase Maven plugin 4.2.0
• Eclipse 20-12 (4.18.0)
• Ubuntu 20.10

A few "random" issues after updates for Microk8s, SpringBoot, SpringCloud, JKube has required some minor tweaks to settings.  I'm still working out improvements for a few settings.

Note that Microk8s is installed via Snap.  Snap is ok but I prefer a bit more control over some packages. I stopped using the snap installed version of Eclipse because of some issues I had run into a while back - seems improved by a local install of Eclipse.  Note that Ubuntu 20.10 isn't an LTS version and that also causes some headaches.  I've had to fight some overall system issues off and on because of my dual graphics card setup (on-chip Intel UHD 630 and a discrete AMD Radeon card). I think this is working ok now but I'm a little disappointed in the setup - I may document that in a different post.

I finally broke down and moved my home directory to an nvme based drive (from standard hard disk) to speed up some activities. That was actually a bit of a painful move - I didn't remember adding an ACL to my home directory. I'd like to move other a few other items to an SSD but I am taking more care because I read some articles indicating that there might be some dependencies across some of the directories/items I'd like to move. The move to the NVME drive drastically helps Eclipse which I  installed off my home directory.

At the service level of my simple SpringBoot based service, I am removing the paging support I started to implement for now. I am still trying to find time to work on this overall and want to have an equivalent Quarkus based service for comparison. I ended up reworking some of the annotations related to using UUID for primary keys since Eclipse started flagging some errors after something changed recently.  I believe that issue is resolved for now. 

A few quick notes on performance. I tend to watch 'top' quite a bit to see what is using resources and also keep and eye on logs in /var/log/.  One issue I fought for a while was logging.  In some cases, some of the services for Microk8s end up producing substantial log out.  In some cases, it is all trace/debug/info. I updated Journald to only log at the warning level - this brings the average system load down nicely.  I also ran across issues with Gnome causing excessive logging (about errors).  I found a reported issue with a patch which fixed it and reduced the logging and overall system load a bit. At this point, kube-apiserver tends to run above 10% CPU on average and I'd like to understand why and whether there is a way to reduce that. I'm also watching for "parasitic" losses from some items I don't need/want.  After the Ubuntu 20.10 upgrade, I found a few services that load but either don't run or they produce needless error logs.  "Masking" those services so they won't/can't start has helped with the parasitic losses.

That's all the time I have for update now.  My regular work and the multitude of home projects are eating up most time now.

Local Development, Microk8s Built-in registry and DNS Parity - Part 1

When I do development at home, I prefer processes that feel "Production Like" in some sense.  For instance, a preference for DNS names in place of IP addresses in configuration data. 

I've been working with Microk8s for a while now at home and I decided to use the local registry it offers as an option.  Why? Well, I want to mimic remote registry functionality and also enable working with some other non-docker container solutions.  I'm checking out different container technologies and the built-in registry uses a newer Containerd version than available in the public install of Docker-CE. I want to analyze the benefits promoted by the newer version a bit. I'm also looking at how similar/seamless a IDE based local deploy of a SpringBoot based service can be compared to deploying into Microk8s.

This post mentions some general ideas I experimented with - there are other solutions that are possibly better and easier. This path helped me learn a few things I might not have otherwise though.

The Microk8s built-in registry documentation expects you to use a convention along the lines of:

    localhost:32000/<appname>:registry

which works fine... but I want to mimic remote registry behavior a bit closer.  I really prefer a scheme like one of the following: 

• docker-star-cases.starcases.com:32000/starcases/<appname>:<version>
• docker-star-cases/starcases/<appname>:<version>

where "star cases" is a fun way I represent my family using the initials of our names.  These mimic the general convention <registry>/<user>/<appname>:<version> where <user> is often an individual user, organization name,  team name, etc.

There are some questions that this goal creates directly related to the registry itself.

1. How do you get the image into the Microk8s built-in registry?
2. How are image references outside of Microk8s resolved?
3. How are image references in deployment descriptors resolved inside Microk8s?

Those questions tend to equate to "How do I resolve a name to a resource identifier (IP or DNS name) from different contexts?".  The 2 initial contexts are:

• Local host (workstation)
• Inside a Microk8s node

There are at least a few ways to do this.  Fundamentally, I want to alias the built-in registry with a name I chose but I want to avoid modifying the existing service as much as possible.

Note that K8s service DNS aliases came up at one time but lack of agreement seemingly killed the idea.

https://github.com/kubernetes/kubernetes/issues/39792

Some general methods available include:

• Editing workstation /etc/hosts to map 'docker-star-cases' to an appropriate IP
• At the home router level, create a static DNS record for 'docker-star-cases' with an appropriate IP
• Possibly using the External DNS module to expose the names outside of Microk8s
• Expose CoreDNS somehow and implement some sort of split horizon

For now, I picked a bit of a compromise between functionality and complexity.  The simplest method would involve updating the /etc/hosts file to provide DNS name to IP mappings for use outside of the Microk8s cluster.  I decided that was too limiting and wanted to stretch the idea a bit further. I really wanted integration with my home DNS.

I'll admit that I did take the easy road here for now.  In my home router, I'm able to add user defined DNS mappings. In this case, I added DNS names of "docker-star-cases" and "docker-star-cases.star-cases.com" with a private IP of 192.x.x.x.. Now, from a terminal window; "nslookup docker-star-cases" returns that private IP address. Accessing the registry didn't work immediately because the registry is setup as insecure and my docker tooling needs configuration to tell it that the insecure registry is ok to use.

The Microk8s instructions mention this but they use their default naming proposal.  So instead of "localhost", I edited "/etc/docker/daemon.json" and added:

{

            "insecure-registries" : ["docker-star-cases:32000"]

         }

and restarted the docker daemon as the Microk8s instructions indicate:

   sudo systemctl restart docker

So at this point; I can generate and tag images with something that resolves to my system using the naming I desired.  The result looks like:

docker-star-cases.starcases.com:32000/starcases/sb-note:latest

and in my local Docker cache (not Microk8s registry) I find:

docker-star-cases.starcases.com:32000/starcases/sb-note   latest                  7a9e91bdf139   41 years ago    311MB

I'm not advocating using 'latest' here but it works well for the example. Note that the "41 years" is related to using JKube.  

The program I am working on is a service implemented by a SpringBoot app which uses (the now) org.eclipse.jkube:kubernetes-maven-plugin to support generating the required kubernetes artifacts and support deployment. The services are just various ideas I'm testing out in general. The service will use a PostgreSQL database.

See 'Prep work' at end of this post for more info on the DB setup.

For now, I'm just using the default namespace. It will probably make sense to utilize other namespaces as needs grow. 

Note that I have a service setup for the DB - the intent is to use that DB for local dev/testing using my IDE and also by the same service deployed within the k8s cluster. This will come up again later.

NAME                                       READY   STATUS    RESTARTS   AGE

pod/springboot-note-db1-57dd9588c4-6x822   1/1     Running   0          2m34s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE

service/kubernetes                ClusterIP   10.152.183.1    <none>        443/TCP    77m

service/springboot-note-db1-svc   ClusterIP   10.152.183.50   <none>        5944/TCP   46m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/springboot-note-db1   1/1     1            1           2m35s

NAME                                             DESIRED   CURRENT   READY   AGE

replicaset.apps/springboot-note-db1-57dd9588c4   1         1         1       2m34s

Using Eclipse and Jkube, I can create and deploy my service now - but that doesn't mean it runs yet. 

[INFO] --- kubernetes-maven-plugin:1.1.0:apply (default-cli) @ starcases-sb-note ---

[INFO] k8s: Using Kubernetes at https://kubernetes.default.svc:16443/ in namespace default with manifest /home/scott/src/eclipse-workspace/SpringBootNote/target/classes/META-INF/jkube/kubernetes.yml 

[INFO] k8s: Updating Service from kubernetes.yml

[INFO] k8s: Updated Service: target/jkube/applyJson/default/service-springboot-note-db1-svc.json

[INFO] k8s: Creating a Service from kubernetes.yml namespace default name starcases-sb-note

[INFO] k8s: Created Service: target/jkube/applyJson/default/service-starcases-sb-note.json

[INFO] k8s: Updating ConfigMap from kubernetes.yml

[INFO] k8s: Updated ConfigMap: target/jkube/applyJson/default/configmap-springboot-note-db1.json

[INFO] k8s: Updating ConfigMap from kubernetes.yml

[INFO] k8s: Updated ConfigMap: target/jkube/applyJson/default/configmap-starcases-sb-note.json

[INFO] k8s: Creating a Deployment from kubernetes.yml namespace default name starcases-sb-note

[INFO] k8s: Created Deployment: target/jkube/applyJson/default/deployment-starcases-sb-note.json

[INFO] k8s: HINT: Use the command `kubectl get pods -w` to watch your pods start up

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

Outside of Microk8s; I can see the image in the registry

$  curl http://docker-star-cases.starcases.com:32000/v2/_catalog

{"repositories":["starcases/sb-note"]}

Here is the microk8s listing after the build/deploy from Eclipse for the service.

NAME                                       READY   STATUS             RESTARTS   AGE

pod/springboot-note-db1-57dd9588c4-6x822   1/1     Running            0          75m

pod/sb-note-8874fcfb-lltw5                 0/1     ImagePullBackOff   0          57m

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

service/kubernetes                ClusterIP   10.152.183.1     <none>        443/TCP    150m

service/springboot-note-db1-svc   ClusterIP   10.152.183.50    <none>        5944/TCP   119m

service/sb-note                   ClusterIP   10.152.183.102   <none>        8080/TCP   57m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/springboot-note-db1   1/1     1            1           75m

deployment.apps/sb-note               0/1     1            0           57m

NAME                                             DESIRED   CURRENT   READY   AGE

replicaset.apps/springboot-note-db1-57dd9588c4   1         1         1       75m

replicaset.apps/sb-note-8874fcfb                 1         1         0       57m

It deployed but doesn't run - the POD logs show that it failed to pull the image.

$ kubectl logs starcases-sb-note-8874fcfb-lltw5

Error from server (BadRequest): container "spring-boot" in pod "sb-note-8874fcfb-lltw5" is waiting to start: trying and failing to pull image

Since we are really trying to treat this as a private registry, if you review the Microk8s documentation for private registries you'll find some setup is needed to allow this new named system to act as a registry mirror. An addition is needed in the file:

    /var/snap/microk8s/current/args/containerd-template.toml

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker-star-cases.starcases.com:32000"]

        endpoint = ["http://docker-star-cases.starcases.com:32000"]

At this point, you need to perform a stop/start on the Microk8s cluster.

$ microk8s stop; microk8s start

And if you check pods now you will find that it did start.

    default              pod/starcases-sb-note-8874fcfb-6g2sr          1/1     Running   1          75s

If you check the POD logs you will find a problem though.  The log entry of interest is:

2021-02-18 06:10:04.108  WARN 1 --- [           main] s.c.k.f.c.Fabric8ConfigMapPropertySource : Can't read configMap with name: [starcases-sb-note] in namespace:[default]. Ignoring.

What?  Did I mention that I want the majority of the service configuration to reside in a config map (and/or secret later) regardless of whether the service is running in the Microk8s cluster or from my Eclipse IDE?  This allows me to work with just one primary source of configuration and allows for some overrides if needed.  We'll, the issue with the service startup is that it was unable to find the configmap and therefore the database connection info was unavailable.

I'll continue this in a Part 2 post to demonstrate a method to get it working and some other details.

[Edit 2021/02/15] 

• NOTES
• I had issues with 'microk8s status' telling me that it wasn't running even though it would show the services up.  This may be due to me creating a soft link to  /snap/bin/microk8s.kubectl using the name kubectl.  I also had an alias of 'kubectl=microk8s kubectl'. 
• Prep Work
• Initial Microk8s initial install / config.
• sudo snap install microk8s --classic

• microk8s enable dns 
• microk8s enable registry storage
• mkdir ~/.kube
• NOTE: Backup any existing ~./.kube/config if needed!
• microk8s config > ~/.kube/config
• Database config map and deployment yamls
• sprngboot-note-db1-configmap.yaml

• apiVersion: v1
  kind: ConfigMap
  metadata:
    name: springboot-note-db1
  data:
    POSTGRES_PASSWORD: PASSWORD 

• springboot-note-db1-deployment.yaml

• kind: Service

  apiVersion: v1

  metadata:

    name:  springboot-note-db1-svc

  spec:

    ports:

      - port: 5944

        targetPort: 5432

        name:  springboot-note-db1-svc

        protocol: TCP

    selector:

      app: springboot-note-db1

  ---

  kind: Deployment

  apiVersion: apps/v1

  metadata:

    annotations:

      configmap.jkube.io/update-on-change: springboot-note-db1

    name: springboot-note-db1

  spec:

    replicas: 1

    selector:

       matchLabels:

          app: springboot-note-db1

    template:

      metadata:

          labels:

            app: springboot-note-db1

      spec:

          containers:

            - name: springboot-note-db1

              image: postgis/postgis:13-3.1

              ports:

                - containerPort: 5432

              envFrom:

                - configMapRef:

                   name: springboot-note-db1

• Applying the settings 
• kubectl apply -f sprngboot-note-db1-configmap.yaml
• kubectl apply -f springboot-note-db1-deployment.yaml

[edit 2021/04/04] Note that wanting to resolve a 1 part name (docker-star-cases) to docker-star-cases.star-cases.com (and then to the final IP address) likely requires setting the search domain.  i.e. on Ubunto 20.10, I: 

• sudo vi /etc/systemd/resolved.conf

and I edit it so under the [Resolve] section, there is a line of:

Domains=star-cases.com