From Microk8s to Usernetes - saga began

 I had a desire to work with more of the "rootless" technologies and found Microk8s a bit limiting.  So I started to look around at what else existed and what state of maturity everything was in.  From a maturity level, much of the work is still pretty new but after looking over information on a few projects, I decided to switch over to Usernetes.

If you review the page you will find a few alternatives and some of the various trade-offs.

The main options listed at the time were Kind, Minikube, Usernetes and K3s with experimental support.

My main initial draw to Usernetes is that it is the reference Kubernetes distribution for Rootless mode.

By this point, I had made some attempts to converge some of the various rootless related setup on my system and had broken some things pretty badly at times while trying to workout changes.  I finally just uninstalled Microk8s and started from that point.  

I was hoping I could still get my deployments from Eclipse working (including running my own private registry) and also keep the SpringBoot / ConfigMap integrations working.

Some of the initial challenges to plan for:

• Microk8s had some features/add-on options you can install which I needed to account for - the major ones being.
• DNS
• dashboard
• Ingress (Nginx)
• MetalLB
• Registry 
• Local Storage

The good news is;

•  DNS wasn't much of a problem.  
• I've not gotten to the dashboard yet.  
• Since I made a conscious choice to avoid Nginx as Ingress in Microk8s - it was less of an issue.  Really, the initial issue was how to avoid the default install of Traefik.  Once I figured that out, things improved a bit.
• I installed HAProxy/kubernetes-ingress just like I had in Microk8s. I still may need to work out some changes to settings yet - not quite clear yet.
  
• MetalLB installed ok - I've still got some items to sort out regarding accessing services from outside the cluster though.
• Helm3 installed without many issues.
• I was able to install cert-manager OK.
• I was able to get the registry installed but  am working through access issues from outside the cluster still.
• I was able to get my image for ldap up and running with some effort.
• I was able to get my Postgresql DB up and  running with some effort

Getting my LDAP and Postgresql images running took unexpected time. I made the conscious decision to work with cgroupv2 when I still Microk8s up but once I got the base Usernetes up, I found issues with accessing existing persistent volumes. The workaround for now was to use setfacl to set permissions in a few cases along with some chown / chmod work and some cluster restarts and other random bits which finally worked out.  I've got some more reading to do on cleaner ways to handle this but I got the images running at least.

I am still working out cluster external access to services - before I didn't have much trouble accessing the LDAP, Postgresql, and registry running in the cluster. Now I've got some extra hoops to sort out for external cluster access. It may come down to just needing calls to:

~/sw/usernetes/rootlessctl.sh add-ports

but I'm not 100% sure yet.  I'm still a bit early in the process.  

Also note that, just to different, I did the Usernetes install for CRIO instead of containerd.

    $ ./install.sh --cri=crio

Which may result in some differences too.

As it stands right now though; I seem to be on the right track:

scott@scott-z1:~$  systemctl --user status -T u7s.target
● u7s.target - Usernetes target (all components in the single node)
     Loaded: loaded (/home/scott/.config/systemd/user/u7s.target; enabled; vendor preset: enabled)
     Active: active since Fri 2021-09-24 08:57:08 EDT; 3 days ago

Sep 24 08:57:08 scott-z1 systemd[2398]: Reached target Usernetes target (all components in the single node).
 

My Podman and Docker are both running rootless at the moment as well but I've got various things tied to containerd. I'd like to simply that a bit more.

But as is, the cluster isn't taking up unrealistic resources.  Working to get logging reasonable with Microk8s and trying to simplify down to an IP4 versus dual stack has helped to a large degree.  A number of services generate quite a bit of logging related to the firewall. I've reduced that a little bit but still some work to go.

I'll have to write up some updates when I get the remaining issues smoothed out - hard to make the time with other commitments though.

Wishing you a blessed day!

Scott

[Update 2022/02/02] I've given up on Usernetes / k3s rootless for now.  The reason is only really related to the usermode networking - slirp4netns.  Maybe this will become easier to use over time in larger deployments but for now it seems a bit of hassle to integrate with existing technology, tools and designs.

Microk8s - minor anoyance

I've been working with Microk8s for a little while now and it generally works well. There is one area where it seems to fall short though - and that is in the area of configuration flexibility. 

I've been starting to work with other tools such as Podman, Buildah, Docker and rootless mode.  I've also been investigating various OCI & CRI runtimes - crun, runc, containerd, cri-o.  As part of that, I want to try the various configurations to see the differences myself.  

Kubernetes deprecated Docker as the runtime for versions 1.21+ and will use CRI in the form of containerd.  This is fine but I would also like to work with crun to determine if there are benefits for some use cases.  Most information I've run across indicates that crun requires less memory and tends to support new features first.  In the use case where I would like to run a good number of services on either a single node cluster or very small cluster as part of some research - crun might provide some benefits.  At the moment, It doesn't seem like configuring Microk8s with crun is an option.  Since OCI and CRI tend to promote the fact that alternative runtimes are supported - I'm somewhat surprised that configuring alternatives in Microk8s is either not supported or possibly just difficult.

I've not found a lot of good/detailed "how-to" articles regarding alternative configuration of many of the container/runtime related technologies. Regardless, I found this post indicating Microk8s lacks the ability to handle alternative setups but the post is old enough that something could have changed or at least the underlying reason may have.

This is a good reason to review other options at least.  I'd rather be able to decide certain tradeoffs based on my needs rather than not have the option. With all the flexibility that K8S provides in general, it seems unfortunate that individual "flavors" may limit configurations. I can see defaulting to some specific configurations - that provides a way to reduce initial complexity but removing/invalidating some configurations seems a bit too "embrace and extend" and/or "Either accept the propriety platform or don't use it" attitudish. I'm hoping I am wrong and can eventually find a way to test out some other configurations of interest.

John 17:3

World English Bible

³ This is eternal life, that they should know you, the only true God, and him whom you sent, Jesus Christ.

Luxery Vinyl Tile - trimming the master bedroom doorways

I've got the entry into the master bedroom, entry to master bath and entry to lanai to do some sort of final trim out.

 I'll start with the master batch entry since that is looking the most tedious (and was).  I'd hate to estimate how long it took to do the 3 pieces of LVT and add some additional porcelain tile in the bathroom doorway.  

I'm going to leave a 1/4" gap and try to caulk it nicely as a first attempt (but not last caulking attempt).

Here is the additional 2 rows dry fitted in the doorway.

 

I still had some pre-mixed mortar which surprissingly turned out still usable.

 

And I had a mostly full bag of dry grout left was well from the bathroom install.

 

Here I have it mortared in.

 

And now it's grouted.

 

And the last 3 pieces of LVT cut to size and trimmed to have an even edge along the porcelin tile.

 

I bought a tube of Keracaulk.U - Biscuit which on the tube looked like a good match but when used it was WAY to light.  Now I'm trying to figure out what color/type caulk to use in this last seam.  I know some folks recommend grouting it instead but I'm not sure how much expansion I might see with the LVT even though it is glued down. I can always go back and put in a T-moulding if this turns out to be a poor choice.

[edit 2021/5/20] I had ended up finding some gray caulk which matched. Looks better but wish I had filled it a bit higher.

The final result for the bedroom/bathroom transition.

For the entry into the master bedroom, I'm taking the original molding and just adapting it a bit to work better with the LVT.  Originally, the carpet side of it was a vertical of ~ 3/4" which is way too tall - someone would be tripping over it. Here is the before picture.

 

Here is an after picture of running it through the router table with a 45 degree bit. I'm hoping this will reduce the chance of catching a toe on it.  I sanded it up and just need to find some new stain of a similar color.

 

 

I had to experiment with stains a bit but I think it came out ok.

 

 

[2021/05/21] And the final doorway - to the lanai.  Still thinking on it.

 I laid some of the LVT under it for now but I'm still working out a final plan.

 

I'll probably end up getting a low threshold and maybe something to trim out the outside edge a bit.