When I do development at home, I prefer processes that feel "Production Like" in some sense. For instance, a preference for DNS names in place of IP addresses in configuration data.
I've been working with Microk8s for a while now at home and I decided to use the local registry it offers as an option. Why? Well, I want to mimic remote registry functionality and also enable working with some other non-docker container solutions. I'm checking out different container technologies and the built-in registry uses a newer Containerd version than available in the public install of Docker-CE. I want to analyze the benefits promoted by the newer version a bit. I'm also looking at how similar/seamless a IDE based local deploy of a SpringBoot based service can be compared to deploying into Microk8s.
This post mentions some general ideas I experimented with - there are other solutions that are possibly better and easier. This path helped me learn a few things I might not have otherwise though.
The Microk8s built-in registry documentation expects you to use a convention along the lines of:
localhost:32000/<appname>:registry
which works fine... but I want to mimic remote registry behavior a bit closer. I really prefer a scheme like one of the following:
- docker-star-cases.starcases.com:32000/starcases/<appname>:<version>
- docker-star-cases/starcases/<appname>:<version>
where "star cases" is a fun way I represent my family using the initials of our names. These mimic the general convention <registry>/<user>/<appname>:<version> where <user> is often an individual user, organization name, team name, etc.
There are some questions that this goal creates directly related to the registry itself.
- How do you get the image into the Microk8s built-in registry?
- How are image references outside of Microk8s resolved?
- How are image references in deployment descriptors resolved inside Microk8s?
Those questions tend to equate to "How do I resolve a name to a resource identifier (IP or DNS name) from different contexts?". The 2 initial contexts are:
- Local host (workstation)
- Inside a Microk8s node
There are at least a few ways to do this. Fundamentally, I want to alias the built-in registry with a name I chose but I want to avoid modifying the existing service as much as possible.
Note that K8s service DNS aliases came up at one time but lack of agreement seemingly killed the idea.
https://github.com/kubernetes/kubernetes/issues/39792
Some general methods available include:
- Editing workstation /etc/hosts to map 'docker-star-cases' to an appropriate IP
- At the home router level, create a static DNS record for 'docker-star-cases' with an appropriate IP
- Possibly using the External DNS module to expose the names outside of Microk8s
- Expose CoreDNS somehow and implement some sort of split horizon
For now, I picked a bit of a compromise between functionality and complexity. The simplest method would involve updating the /etc/hosts file to provide DNS name to IP mappings for use outside of the Microk8s cluster. I decided that was too limiting and wanted to stretch the idea a bit further. I really wanted integration with my home DNS.
I'll admit that I did take the easy road here for now. In my home router, I'm able to add user defined DNS mappings. In this case, I added DNS names of "docker-star-cases" and "docker-star-cases.star-cases.com" with a private IP of 192.x.x.x.. Now, from a terminal window; "nslookup docker-star-cases" returns that private IP address. Accessing the registry didn't work immediately because the registry is setup as insecure and my docker tooling needs configuration to tell it that the insecure registry is ok to use.
The Microk8s instructions mention this but they use their default naming proposal. So instead of "localhost", I edited "/etc/docker/daemon.json" and added:
{
"insecure-registries" : ["docker-star-cases:32000"]
}
and restarted the docker daemon as the Microk8s instructions indicate:
sudo systemctl restart docker
So at this point; I can generate and tag images with something that resolves to my system using the naming I desired. The result looks like:
docker-star-cases.starcases.com:32000/starcases/sb-note:latest
and in my local Docker cache (not Microk8s registry) I find:
docker-star-cases.starcases.com:32000/starcases/sb-note latest 7a9e91bdf139 41 years ago 311MB
I'm not advocating using 'latest' here but it works well for the example. Note that the "41 years" is related to using JKube.
The program I am working on is a service implemented by a SpringBoot app which uses (the now) org.eclipse.jkube:kubernetes-maven-plugin to support generating the required kubernetes artifacts and support deployment. The services are just various ideas I'm testing out in general. The service will use a PostgreSQL database.
See 'Prep work' at end of this post for more info on the DB setup.
For now, I'm just using the default namespace. It will probably make sense to utilize other namespaces as needs grow.
Note that I have a service setup for the DB - the intent is to use that DB for local dev/testing using my IDE and also by the same service deployed within the k8s cluster. This will come up again later.
NAME READY STATUS RESTARTS AGE
pod/springboot-note-db1-57dd9588c4-6x822 1/1 Running 0 2m34s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 77m
service/springboot-note-db1-svc ClusterIP 10.152.183.50 <none> 5944/TCP 46m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/springboot-note-db1 1/1 1 1 2m35s
NAME DESIRED CURRENT READY AGE
replicaset.apps/springboot-note-db1-57dd9588c4 1 1 1 2m34s
Using Eclipse and Jkube, I can create and deploy my service now - but that doesn't mean it runs yet.
[INFO] --- kubernetes-maven-plugin:1.1.0:apply (default-cli) @ starcases-sb-note ---
[INFO] k8s: Using Kubernetes at https://kubernetes.default.svc:16443/ in namespace default with manifest /home/scott/src/eclipse-workspace/SpringBootNote/target/classes/META-INF/jkube/kubernetes.yml
[INFO] k8s: Updating Service from kubernetes.yml
[INFO] k8s: Updated Service: target/jkube/applyJson/default/service-springboot-note-db1-svc.json
[INFO] k8s: Creating a Service from kubernetes.yml namespace default name starcases-sb-note
[INFO] k8s: Created Service: target/jkube/applyJson/default/service-starcases-sb-note.json
[INFO] k8s: Updating ConfigMap from kubernetes.yml
[INFO] k8s: Updated ConfigMap: target/jkube/applyJson/default/configmap-springboot-note-db1.json
[INFO] k8s: Updating ConfigMap from kubernetes.yml
[INFO] k8s: Updated ConfigMap: target/jkube/applyJson/default/configmap-starcases-sb-note.json
[INFO] k8s: Creating a Deployment from kubernetes.yml namespace default name starcases-sb-note
[INFO] k8s: Created Deployment: target/jkube/applyJson/default/deployment-starcases-sb-note.json
[INFO] k8s: HINT: Use the command `kubectl get pods -w` to watch your pods start up
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
Outside of Microk8s; I can see the image in the registry
$ curl http://docker-star-cases.starcases.com:32000/v2/_catalog
{"repositories":["starcases/sb-note"]}
Here is the microk8s listing after the build/deploy from Eclipse for the service.
NAME READY STATUS RESTARTS AGE
pod/springboot-note-db1-57dd9588c4-6x822 1/1 Running 0 75m
pod/sb-note-8874fcfb-lltw5 0/1 ImagePullBackOff 0 57m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 150m
service/springboot-note-db1-svc ClusterIP 10.152.183.50 <none> 5944/TCP 119m
service/sb-note ClusterIP 10.152.183.102 <none> 8080/TCP 57m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/springboot-note-db1 1/1 1 1 75m
deployment.apps/sb-note 0/1 1 0 57m
NAME DESIRED CURRENT READY AGE
replicaset.apps/springboot-note-db1-57dd9588c4 1 1 1 75m
replicaset.apps/sb-note-8874fcfb 1 1 0 57m
It deployed but doesn't run - the POD logs show that it failed to pull the image.
$ kubectl logs starcases-sb-note-8874fcfb-lltw5
Error from server (BadRequest): container "spring-boot" in pod "sb-note-8874fcfb-lltw5" is waiting to start: trying and failing to pull image
Since we are really trying to treat this as a private registry, if you review the Microk8s documentation for private registries you'll find some setup is needed to allow this new named system to act as a registry mirror. An addition is needed in the file:
/var/snap/microk8s/current/args/containerd-template.toml
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker-star-cases.starcases.com:32000"]
endpoint = ["http://docker-star-cases.starcases.com:32000"]
At this point, you need to perform a stop/start on the Microk8s cluster.
$ microk8s stop; microk8s start
And if you check pods now you will find that it did start.
default pod/starcases-sb-note-8874fcfb-6g2sr 1/1 Running 1 75s
If you check the POD logs you will find a problem though. The log entry of interest is:
2021-02-18 06:10:04.108 WARN 1 --- [ main] s.c.k.f.c.Fabric8ConfigMapPropertySource : Can't read configMap with name: [starcases-sb-note] in namespace:[default]. Ignoring.
What? Did I mention that I want the majority of the service configuration to reside in a config map (and/or secret later) regardless of whether the service is running in the Microk8s cluster or from my Eclipse IDE? This allows me to work with just one primary source of configuration and allows for some overrides if needed. We'll, the issue with the service startup is that it was unable to find the configmap and therefore the database connection info was unavailable.
I'll continue this in a Part 2 post to demonstrate a method to get it working and some other details.
[Edit 2021/02/15]
- NOTES
- I had issues with 'microk8s status' telling me that it wasn't running even though it would show the services up. This may be due to me creating a soft link to /snap/bin/microk8s.kubectl using the name kubectl. I also had an alias of 'kubectl=microk8s kubectl'.
- Prep Work
- Initial Microk8s initial install / config.
- sudo snap install microk8s --classic
- microk8s enable dns
- microk8s enable registry storage
- mkdir ~/.kube
- NOTE: Backup any existing ~./.kube/config if needed!
- microk8s config > ~/.kube/config
- Database config map and deployment yamls
- sprngboot-note-db1-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: springboot-note-db1
data:
POSTGRES_PASSWORD: PASSWORD
- springboot-note-db1-deployment.yaml
kind: Service
apiVersion: v1
metadata:
name: springboot-note-db1-svc
spec:
ports:
- port: 5944
targetPort: 5432
name: springboot-note-db1-svc
protocol: TCP
selector:
app: springboot-note-db1
---
kind: Deployment
apiVersion: apps/v1
metadata:
annotations:
configmap.jkube.io/update-on-change: springboot-note-db1
name: springboot-note-db1
spec:
replicas: 1
selector:
matchLabels:
app: springboot-note-db1
template:
metadata:
labels:
app: springboot-note-db1
spec:
containers:
- name: springboot-note-db1
image: postgis/postgis:13-3.1
ports:
- containerPort: 5432
envFrom:
- configMapRef:
name: springboot-note-db1
- Applying the settings
- kubectl apply -f sprngboot-note-db1-configmap.yaml
- kubectl apply -f springboot-note-db1-deployment.yaml
[edit 2021/04/04] Note that wanting to resolve a 1 part name (docker-star-cases) to docker-star-cases.star-cases.com (and then to the final IP address) likely requires setting the search domain. i.e. on Ubunto 20.10, I:
- sudo vi /etc/systemd/resolved.conf
and I edit it so under the [Resolve] section, there is a line of:
Domains=star-cases.com
