In part 2 of this, I had a SpringBoot service which was runnable from both an IDE and from Microk8s - with the configuration for both methods coming from a configmap in Microkk8s.
Here I want to go into the configuration a bit more because that part created a few headaches but I think it is sorted out now (for the most part).
To start with, one of the major enablers for this setup is a bootstrap.yml file. In this case, I'm not using any application.properties file in the apps main/resources folder. The use of the bootstrap file is to get required settings, well - bootstrapped before the majority of Spring auto-configuration starts up.
Spring profiles are also a key part of this setup. The main spring config in the bootstrap file is defined for the profiles:
I'll admit that I should have documented this better initially - a key aspect is the kubernetes profile though. There are likely other ways to get things working but so far this is cleanest and most reliable. Note that the kubernetes profile is used even when executing the service in the IDE. I'll explain that a bit more in minute but first lets look at the initial part of the bootstrap configuration.
spring:
config:
activate:
on-profile:
- native
- default
- kubernetes
application:
name: sb-note
cloud:
discovery:
enabled: true
kubernetes:
enabled: true
config:
enabled: true
enable-api: true
paths:
- /deployments/config/application.properties
discovery:
enabled: true
client:
namespace: default
clientCertFile: /var/snap/microk8s/current/certs/kubelet.crt
caCertFile: /var/snap/microk8s/current/certs/kubelet.crt
ca-cert-file: /var/snap/microk8s/current/certs/kubelet.crt
trust-certs: true
masterUrl: https://kubernetes:443
springdoc:
swagger-ui.path: /swagger-ui.html
debug: true
Note that the various cloud/kubernetes aspects are enabled. These are important for the dependencies:
- org.springframework.cloud:spring-cloud-starter-bootstrap
- org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8-config
- org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8
- org.springframework.cloud:spring-cloud-starter-kubernetes-fabric8-all
I may be able to simplify those a bit but that is what I have after thrashing a bit with a recent SpringCloud update. The combination provides the ability to "know about" kubernetes and use features like configmaps. Since I always want to pull my main configuration using a configmap, I always use the kubernetes profile and the enablements above so configmaps are usable.
Note that the "client" section is what generally provides any configuration needed so your app can leverage the "kubernetes" awareness and access those associated resources. Note that the client.masterUrl though is defined in the manner which the service needs it when it actually is running in Microk8s. The https://kubernetes:443 is not accessible from outside of Microk8s. So this config is all I needed for running in Kubernetes - but what about when running it from outside?
Well, there is a second part to the above bootstrap.yml file which is:
---
spring:
config:
activate:
on-profile:
- ide
cloud:
kubernetes:
client:
masterUrl: https://kubernetes.default.svc:16443
and when I run the service in the IDE, I add an additional spring profile to end of my list of active profiles: ide
The result is that the client.masterUrl of https://kubernetes.default.svc:16443 overrides the previous definition due to ordering. The kubernetes.default.svc is what I have been manually maintaining an IP for in my home routers DNS. That DNS name resolves to the cluster IP of the kubernetes cluster and the port 16443 which is what Microk8s exposes it on - see their documentation on ports.
So this allows the service to see and access kubernetes and the associated resources from both inside and outside of the cluster. Note that I may be able to move some of the other client settings to the "ide" profile but haven't gotten quite that far in my cleanup. I believe that this can be done because I don't think the various paths for the client section are probably visible as shows show - so are probably ignored when running inside kubernetes.
Note, initially I had things working in a somewhat awkward fashion which this setup fixes. If you don't override the masterUrl then you can still make things work but it is odd. Basically, if your deployment exposes the configmap as both a configmap AND an application.properties file - when running in kubernetes and the masterUrl doesn't resolve correctly, the configmap won't be found but the application.properties file is mounted and loaded via normal spring auto-configuration. You'll see some error/warnings in logs but configuration is available. Once I realized that is what I had done, I worked out this additional profile so to prevent that configuration from occurring. The current config shown above doesn't result in those error conditions.
Note that the configmap is defined as the following.
namespace: default
data:
application.properties: |-
spring.profiles.active=kubernetes
debug=false
spring.application=sb-note
spring.datasource.url=jdbc:postgresql://springboot-note-db1-svc.default.svc:5944/postgres
spring.datasource.username=postgres
spring.datasource.password=PASSWORD
spring.jpa.database=POSTGRESQL
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
spring.jpa.hibernate.ddl-auto=update
springdoc.swagger-ui.path=/swagger-ui.html
logging.level.org.hibernate.SQL=DEBUG
logging.level.org.hibernate.type=WARN
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.format_sql=true
javax.persistence.schema-generation.database.action=update
So the configmap data must be available to the service in some form or the database connection initialization will fail. That is the one of the most visible ways you will know that it isn't finding the information - which is true while running in the IDE and in k8s as well. I'll note that I am not using k8s secrets here yet - that would be the next step in doing a more "enterprise" setup. That should work in a similar fashion to the configmap setup.
Note that so far, all this setup is for actual runtime and not deployment - although the above configmap is defined in the way that Jkube needs it so as to create the actual configmap during deployment.
For build/deployment, a few key properties are needed in the maven pom file.
<java.version>11</java.version>
<docker.starcases.registry>docker-star-cases.starcases.com:32000</docker.starcases.registry>
<image.user>starcases</image.user>
<spring-cloud.version>2020.0.1</spring-cloud.version>
<jkube.masterUrl>https://kubernetes.default.svc:16443</jkube.masterUrl>
<jkube.trustStoreFile>/var/snap/microk8s/current/certs/kubelet.crt</jkube.trustStoreFile>
<jkube.version>1.1.1</jkube.version>
<jkube.recreate>true</jkube.recreate>
<jkube.deploy.ignoreServices>true</jkube.deploy.ignoreServices>
<jkube.generator.name>${docker.starcases.registry}/${image.user}/${project.artifactId}:${project.version}</jkube.generator.name>
<jkube-revision-history>1</jkube-revision-history>
Note that I am having springboot build the service and JKube is repackaging into a docker compatible image and then deploying that image to the Microk8s registry and then deploying the other resources Jkube generates configuration for from my setup.
The main build is being done using the recent SpringBoot 2.3 feature enhancements - the pom.xml part of interest is.
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludeDevtools>false</excludeDevtools>
<release>11</release>
<image>
<name>${jkube.generator.name}</name>
</image>
<layers>
<enabled>true</enabled>
<includeLayerTools>true</includeLayerTools>
</layers>
</configuration>
</plugin>
</plugins>
And then rest of the repackaging and deployment is handled by:
<profile>
<id>kubernetes</id>
<build>
<plugins>
<plugin>
<groupId>org.eclipse.jkube</groupId>
<artifactId>kubernetes-maven-plugin</artifactId>
<version>${jkube.version}</version>
<configuration>
<profile>minimal</profile>
<buildStrategy>docker</buildStrategy>
<detail>true</detail>
<imagePullPolicy>Always</imagePullPolicy>
<logStdout>true</logStdout>
<verbose>true</verbose>
<failOnNoKubernetesJson>true</failOnNoKubernetesJson>
<failOnValidationError>true</failOnValidationError>
<skipExtendedAuth>true</skipExtendedAuth>
<pushRegistry>${docker.starcases.registry}</pushRegistry>
<skipBuild>true</skipBuild>
<kubernetesManifest>${basedir}/target/classes/META-INF/jkube/kubernetes.yml</kubernetesManifest>
<resources>
<labels>
<all>
<property>
<name>app</name>
<value>${project.artifactId}</value>
</property>
<property>
<name>provider</name>
<value>jkube</value>
</property>
<property>
<name>group</name>
<value>${project.groupId}</value>
</property>
<property>
<name>version</name>
<value>${project.version}</value>
</property>
</all>
</labels>
</resources>
</configuration>
</plugin>
</plugins>
</build>
</profile>
There are likely a few items I can cleanup here but it is functional.
Note that the maven goal I use in the IDE for the Microk8s build / deployment is:
- clean spring-boot:build-image k8s:resource k8s:push k8s:apply
And I specify a profile of "kubernetes". I do have a few environmental settings as well - maven.test.skip, skip tests, and an item for trusting the kubernetes certificate which I may not need now (at one point I worked out some issues related to it but am noticing I didn't clean all settings in all the IDE launch configs). I'll have to recheck that - at one point I did need to add an additional name to the certificate to get things to match between the inside and outside kubernetes views of the world.
And then note that when launching the service in the IDE (as a SpringBoot app), the profiles that are enabled are: local,default,native,kubernetes,ide
As I've stated a few times, I might be able to simplify that list a little but haven't tried. It grew a bit as I was trying to work with GraalVM and was copying settings back and forth between projects at one point.
Hoping your day is blessed!
Scott
