From Microk8s to Usernetes - saga began

 I had a desire to work with more of the "rootless" technologies and found Microk8s a bit limiting.  So I started to look around at what else existed and what state of maturity everything was in.  From a maturity level, much of the work is still pretty new but after looking over information on a few projects, I decided to switch over to Usernetes.

If you review the page you will find a few alternatives and some of the various trade-offs.

The main options listed at the time were Kind, Minikube, Usernetes and K3s with experimental support.

My main initial draw to Usernetes is that it is the reference Kubernetes distribution for Rootless mode.

By this point, I had made some attempts to converge some of the various rootless related setup on my system and had broken some things pretty badly at times while trying to workout changes.  I finally just uninstalled Microk8s and started from that point.  

I was hoping I could still get my deployments from Eclipse working (including running my own private registry) and also keep the SpringBoot / ConfigMap integrations working.

Some of the initial challenges to plan for:

• Microk8s had some features/add-on options you can install which I needed to account for - the major ones being.
• DNS
• dashboard
• Ingress (Nginx)
• MetalLB
• Registry 
• Local Storage

The good news is;

•  DNS wasn't much of a problem.  
• I've not gotten to the dashboard yet.  
• Since I made a conscious choice to avoid Nginx as Ingress in Microk8s - it was less of an issue.  Really, the initial issue was how to avoid the default install of Traefik.  Once I figured that out, things improved a bit.
• I installed HAProxy/kubernetes-ingress just like I had in Microk8s. I still may need to work out some changes to settings yet - not quite clear yet.
  
• MetalLB installed ok - I've still got some items to sort out regarding accessing services from outside the cluster though.
• Helm3 installed without many issues.
• I was able to install cert-manager OK.
• I was able to get the registry installed but  am working through access issues from outside the cluster still.
• I was able to get my image for ldap up and running with some effort.
• I was able to get my Postgresql DB up and  running with some effort

Getting my LDAP and Postgresql images running took unexpected time. I made the conscious decision to work with cgroupv2 when I still Microk8s up but once I got the base Usernetes up, I found issues with accessing existing persistent volumes. The workaround for now was to use setfacl to set permissions in a few cases along with some chown / chmod work and some cluster restarts and other random bits which finally worked out.  I've got some more reading to do on cleaner ways to handle this but I got the images running at least.

I am still working out cluster external access to services - before I didn't have much trouble accessing the LDAP, Postgresql, and registry running in the cluster. Now I've got some extra hoops to sort out for external cluster access. It may come down to just needing calls to:

~/sw/usernetes/rootlessctl.sh add-ports

but I'm not 100% sure yet.  I'm still a bit early in the process.  

Also note that, just to different, I did the Usernetes install for CRIO instead of containerd.

    $ ./install.sh --cri=crio

Which may result in some differences too.

As it stands right now though; I seem to be on the right track:

scott@scott-z1:~$  systemctl --user status -T u7s.target
● u7s.target - Usernetes target (all components in the single node)
     Loaded: loaded (/home/scott/.config/systemd/user/u7s.target; enabled; vendor preset: enabled)
     Active: active since Fri 2021-09-24 08:57:08 EDT; 3 days ago

Sep 24 08:57:08 scott-z1 systemd[2398]: Reached target Usernetes target (all components in the single node).
 

My Podman and Docker are both running rootless at the moment as well but I've got various things tied to containerd. I'd like to simply that a bit more.

But as is, the cluster isn't taking up unrealistic resources.  Working to get logging reasonable with Microk8s and trying to simplify down to an IP4 versus dual stack has helped to a large degree.  A number of services generate quite a bit of logging related to the firewall. I've reduced that a little bit but still some work to go.

I'll have to write up some updates when I get the remaining issues smoothed out - hard to make the time with other commitments though.

Wishing you a blessed day!

Scott

[Update 2022/02/02] I've given up on Usernetes / k3s rootless for now.  The reason is only really related to the usermode networking - slirp4netns.  Maybe this will become easier to use over time in larger deployments but for now it seems a bit of hassle to integrate with existing technology, tools and designs.