Local Development, Microk8s Built-in registry and DNS Parity - Part 1

When I do development at home, I prefer processes that feel "Production Like" in some sense.  For instance, a preference for DNS names in place of IP addresses in configuration data. 

I've been working with Microk8s for a while now at home and I decided to use the local registry it offers as an option.  Why? Well, I want to mimic remote registry functionality and also enable working with some other non-docker container solutions.  I'm checking out different container technologies and the built-in registry uses a newer Containerd version than available in the public install of Docker-CE. I want to analyze the benefits promoted by the newer version a bit. I'm also looking at how similar/seamless a IDE based local deploy of a SpringBoot based service can be compared to deploying into Microk8s.

This post mentions some general ideas I experimented with - there are other solutions that are possibly better and easier. This path helped me learn a few things I might not have otherwise though.

The Microk8s built-in registry documentation expects you to use a convention along the lines of:

    localhost:32000/<appname>:registry

which works fine... but I want to mimic remote registry behavior a bit closer.  I really prefer a scheme like one of the following: 

• docker-star-cases.starcases.com:32000/starcases/<appname>:<version>
• docker-star-cases/starcases/<appname>:<version>

where "star cases" is a fun way I represent my family using the initials of our names.  These mimic the general convention <registry>/<user>/<appname>:<version> where <user> is often an individual user, organization name,  team name, etc.

There are some questions that this goal creates directly related to the registry itself.

1. How do you get the image into the Microk8s built-in registry?
2. How are image references outside of Microk8s resolved?
3. How are image references in deployment descriptors resolved inside Microk8s?

Those questions tend to equate to "How do I resolve a name to a resource identifier (IP or DNS name) from different contexts?".  The 2 initial contexts are:

• Local host (workstation)
• Inside a Microk8s node

There are at least a few ways to do this.  Fundamentally, I want to alias the built-in registry with a name I chose but I want to avoid modifying the existing service as much as possible.

Note that K8s service DNS aliases came up at one time but lack of agreement seemingly killed the idea.

https://github.com/kubernetes/kubernetes/issues/39792

Some general methods available include:

• Editing workstation /etc/hosts to map 'docker-star-cases' to an appropriate IP
• At the home router level, create a static DNS record for 'docker-star-cases' with an appropriate IP
• Possibly using the External DNS module to expose the names outside of Microk8s
• Expose CoreDNS somehow and implement some sort of split horizon

For now, I picked a bit of a compromise between functionality and complexity.  The simplest method would involve updating the /etc/hosts file to provide DNS name to IP mappings for use outside of the Microk8s cluster.  I decided that was too limiting and wanted to stretch the idea a bit further. I really wanted integration with my home DNS.

I'll admit that I did take the easy road here for now.  In my home router, I'm able to add user defined DNS mappings. In this case, I added DNS names of "docker-star-cases" and "docker-star-cases.star-cases.com" with a private IP of 192.x.x.x.. Now, from a terminal window; "nslookup docker-star-cases" returns that private IP address. Accessing the registry didn't work immediately because the registry is setup as insecure and my docker tooling needs configuration to tell it that the insecure registry is ok to use.

The Microk8s instructions mention this but they use their default naming proposal.  So instead of "localhost", I edited "/etc/docker/daemon.json" and added:

{

            "insecure-registries" : ["docker-star-cases:32000"]

         }

and restarted the docker daemon as the Microk8s instructions indicate:

   sudo systemctl restart docker

So at this point; I can generate and tag images with something that resolves to my system using the naming I desired.  The result looks like:

docker-star-cases.starcases.com:32000/starcases/sb-note:latest

and in my local Docker cache (not Microk8s registry) I find:

docker-star-cases.starcases.com:32000/starcases/sb-note   latest                  7a9e91bdf139   41 years ago    311MB

I'm not advocating using 'latest' here but it works well for the example. Note that the "41 years" is related to using JKube.  

The program I am working on is a service implemented by a SpringBoot app which uses (the now) org.eclipse.jkube:kubernetes-maven-plugin to support generating the required kubernetes artifacts and support deployment. The services are just various ideas I'm testing out in general. The service will use a PostgreSQL database.

See 'Prep work' at end of this post for more info on the DB setup.

For now, I'm just using the default namespace. It will probably make sense to utilize other namespaces as needs grow. 

Note that I have a service setup for the DB - the intent is to use that DB for local dev/testing using my IDE and also by the same service deployed within the k8s cluster. This will come up again later.

NAME                                       READY   STATUS    RESTARTS   AGE

pod/springboot-note-db1-57dd9588c4-6x822   1/1     Running   0          2m34s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE

service/kubernetes                ClusterIP   10.152.183.1    <none>        443/TCP    77m

service/springboot-note-db1-svc   ClusterIP   10.152.183.50   <none>        5944/TCP   46m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/springboot-note-db1   1/1     1            1           2m35s

NAME                                             DESIRED   CURRENT   READY   AGE

replicaset.apps/springboot-note-db1-57dd9588c4   1         1         1       2m34s

Using Eclipse and Jkube, I can create and deploy my service now - but that doesn't mean it runs yet. 

[INFO] --- kubernetes-maven-plugin:1.1.0:apply (default-cli) @ starcases-sb-note ---

[INFO] k8s: Using Kubernetes at https://kubernetes.default.svc:16443/ in namespace default with manifest /home/scott/src/eclipse-workspace/SpringBootNote/target/classes/META-INF/jkube/kubernetes.yml 

[INFO] k8s: Updating Service from kubernetes.yml

[INFO] k8s: Updated Service: target/jkube/applyJson/default/service-springboot-note-db1-svc.json

[INFO] k8s: Creating a Service from kubernetes.yml namespace default name starcases-sb-note

[INFO] k8s: Created Service: target/jkube/applyJson/default/service-starcases-sb-note.json

[INFO] k8s: Updating ConfigMap from kubernetes.yml

[INFO] k8s: Updated ConfigMap: target/jkube/applyJson/default/configmap-springboot-note-db1.json

[INFO] k8s: Updating ConfigMap from kubernetes.yml

[INFO] k8s: Updated ConfigMap: target/jkube/applyJson/default/configmap-starcases-sb-note.json

[INFO] k8s: Creating a Deployment from kubernetes.yml namespace default name starcases-sb-note

[INFO] k8s: Created Deployment: target/jkube/applyJson/default/deployment-starcases-sb-note.json

[INFO] k8s: HINT: Use the command `kubectl get pods -w` to watch your pods start up

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

Outside of Microk8s; I can see the image in the registry

$  curl http://docker-star-cases.starcases.com:32000/v2/_catalog

{"repositories":["starcases/sb-note"]}

Here is the microk8s listing after the build/deploy from Eclipse for the service.

NAME                                       READY   STATUS             RESTARTS   AGE

pod/springboot-note-db1-57dd9588c4-6x822   1/1     Running            0          75m

pod/sb-note-8874fcfb-lltw5                 0/1     ImagePullBackOff   0          57m

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

service/kubernetes                ClusterIP   10.152.183.1     <none>        443/TCP    150m

service/springboot-note-db1-svc   ClusterIP   10.152.183.50    <none>        5944/TCP   119m

service/sb-note                   ClusterIP   10.152.183.102   <none>        8080/TCP   57m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/springboot-note-db1   1/1     1            1           75m

deployment.apps/sb-note               0/1     1            0           57m

NAME                                             DESIRED   CURRENT   READY   AGE

replicaset.apps/springboot-note-db1-57dd9588c4   1         1         1       75m

replicaset.apps/sb-note-8874fcfb                 1         1         0       57m

It deployed but doesn't run - the POD logs show that it failed to pull the image.

$ kubectl logs starcases-sb-note-8874fcfb-lltw5

Error from server (BadRequest): container "spring-boot" in pod "sb-note-8874fcfb-lltw5" is waiting to start: trying and failing to pull image

Since we are really trying to treat this as a private registry, if you review the Microk8s documentation for private registries you'll find some setup is needed to allow this new named system to act as a registry mirror. An addition is needed in the file:

    /var/snap/microk8s/current/args/containerd-template.toml

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker-star-cases.starcases.com:32000"]

        endpoint = ["http://docker-star-cases.starcases.com:32000"]

At this point, you need to perform a stop/start on the Microk8s cluster.

$ microk8s stop; microk8s start

And if you check pods now you will find that it did start.

    default              pod/starcases-sb-note-8874fcfb-6g2sr          1/1     Running   1          75s

If you check the POD logs you will find a problem though.  The log entry of interest is:

2021-02-18 06:10:04.108  WARN 1 --- [           main] s.c.k.f.c.Fabric8ConfigMapPropertySource : Can't read configMap with name: [starcases-sb-note] in namespace:[default]. Ignoring.

What?  Did I mention that I want the majority of the service configuration to reside in a config map (and/or secret later) regardless of whether the service is running in the Microk8s cluster or from my Eclipse IDE?  This allows me to work with just one primary source of configuration and allows for some overrides if needed.  We'll, the issue with the service startup is that it was unable to find the configmap and therefore the database connection info was unavailable.

I'll continue this in a Part 2 post to demonstrate a method to get it working and some other details.

[Edit 2021/02/15] 

• NOTES
• I had issues with 'microk8s status' telling me that it wasn't running even though it would show the services up.  This may be due to me creating a soft link to  /snap/bin/microk8s.kubectl using the name kubectl.  I also had an alias of 'kubectl=microk8s kubectl'. 
• Prep Work
• Initial Microk8s initial install / config.
• sudo snap install microk8s --classic

• microk8s enable dns 
• microk8s enable registry storage
• mkdir ~/.kube
• NOTE: Backup any existing ~./.kube/config if needed!
• microk8s config > ~/.kube/config
• Database config map and deployment yamls
• sprngboot-note-db1-configmap.yaml

• apiVersion: v1
  kind: ConfigMap
  metadata:
    name: springboot-note-db1
  data:
    POSTGRES_PASSWORD: PASSWORD 

• springboot-note-db1-deployment.yaml

• kind: Service

  apiVersion: v1

  metadata:

    name:  springboot-note-db1-svc

  spec:

    ports:

      - port: 5944

        targetPort: 5432

        name:  springboot-note-db1-svc

        protocol: TCP

    selector:

      app: springboot-note-db1

  ---

  kind: Deployment

  apiVersion: apps/v1

  metadata:

    annotations:

      configmap.jkube.io/update-on-change: springboot-note-db1

    name: springboot-note-db1

  spec:

    replicas: 1

    selector:

       matchLabels:

          app: springboot-note-db1

    template:

      metadata:

          labels:

            app: springboot-note-db1

      spec:

          containers:

            - name: springboot-note-db1

              image: postgis/postgis:13-3.1

              ports:

                - containerPort: 5432

              envFrom:

                - configMapRef:

                   name: springboot-note-db1

• Applying the settings 
• kubectl apply -f sprngboot-note-db1-configmap.yaml
• kubectl apply -f springboot-note-db1-deployment.yaml

[edit 2021/04/04] Note that wanting to resolve a 1 part name (docker-star-cases) to docker-star-cases.star-cases.com (and then to the final IP address) likely requires setting the search domain.  i.e. on Ubunto 20.10, I: 

• sudo vi /etc/systemd/resolved.conf

and I edit it so under the [Resolve] section, there is a line of:

Domains=star-cases.com