“I am Overwhelm with Helm” – Kubernetes

Transforming the way people build, test & deliver software

“I am Overwhelm with Helm” – Kubernetes

Two months back when we embarked a journey of collaborating with our Bangalore based fin-tech client to help them transition their VM based deployments to full-fledged container orchestration. With no footprint of the DevOps team at a client firm, we had a task to come up with a solution which is more scalable, easily understandable and most important aspect maintainable. The mix of Java, Ruby and Node services to be orchestrated added a bit of extra spin to the previously mentioned spices of the recipe we were trying to achieve.

We thought of two options listed below to achieve scalability, understandability, and maintainability

Approach 1

  • Write Ansible scripts to execute Kubernetes native commands
  • Use Ansible templates to manage application override configurations.
  • Write Ansible scripts to manage rollbacks and updates.

Approach 2

  • Write Helm charts and not worry about Kubernetes native commands
  • Helm gives out of the box support to templatize Kubernetes resource deployment files.
  • Use Go language templates support to manage application override configurations.
  • Helm gives out of the box support for rollbacks and updates.

Clearly Helm made a cut on the point of scalability and maintainability as we didn’t have to maintain additional updates and rollback scripts. Now, we had to make sure scripts we write are scalable and understandable. We decided on the approach of writing templatized Helm charts for Java, Ruby and Node each, so that to onboard any new service we didn’t have to do many changes. Just add override configuration files, & properties files and off you go.

Benefits

  • New services are ready for deployment within 10 minutes by creating on override property files
  • Development teams are able to contribute to YAML based deployment strategies
  • Clear isolation of environment management
  • Everything is a source controlled
  • Most importantly, our client is happy 😊

Now, let us get into the implementation of deploying a Java-based Springboot application using helm package manager – one of the common questions is, what we need to do to achieve that? Let’s list them out:

  • Containerized the code using Dockerfile (Make sure that packaged code is environment agnostic)
  • Define application.properties/YAML file
  • Identify environment variables that need to be injected into a container
  • Identify volumes(secret volume) that need to be injected into a container

Let’s solve them one by one. How to orchestrate containerized Springboot application

1. Create a Dockerfile

FROM openjdk:8u111-jdk-alpine
ENV MAIN_OPTS '' 
RUN mkdir -p /opt/my-boot-service/ /opt/appConfig
COPY target/my-boot-service.jar /opt/my-boot-service/
WORKDIR /opt/my-boot-service/
ENTRYPOINT java $JAVA_OPTS -jar ./my-boot-service.jar $MAIN_OPTS
EXPOSE 8080

2. Define application.properties

Create an environment-specific application.properties file using the below folder structure.
${ROOTDIR}/namespace/<ENV>/<application_name>/config/application.properties
It will help bifurcate environment-specific files with clear isolation.

3. Create secrets.yaml for Kubernetes secret resource

{{- if .Values.imageCredentials}}
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.imageCredentials.secretName }}
  namespace: {{ .Values.namespace}}
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: {{ template "imagePullSecret" . }}
{{- end}}
--- 
# Based on namespace from values.yaml we can define what files to put into secrets.
# Adding volume secret to avoid any confusion with envsecrets (IF any)
{{- $path := printf "namespaces/%s/%s/configs/*" .Values.namespace .Values.appName  }}
{{- if $path }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.appName }}-volume-sec
  namespace: {{ .Values.namespace}}
  labels:
    app: {{ .Values.appName }}
    chart: "{{ .Values.appName }}-{{ .Chart.Version | replace "+" "_" }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
{{ (.Files.Glob $path).AsSecrets | indent 2 }}
{{- end }}
--- 
# Setup env secrets 
{{- if .Values.env.secrets }}
{{- $root := .Values.env.secrets }}
apiVersion: v1f
kind: Secret
metadata:
  name: {{ .Values.appName }}-env-secret
  namespace: {{ .Values.namespace}}
  labels:
    app: {{ .Values.appName }}
    chart: "{{ .Values.appName }}-{{ .Chart.Version | replace "+" "_" }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
{{- range $k,$v := $root }}
  {{$k }}: {{ default "" $v | b64enc | quote }}
{{- end}}
{{- end }}

4. Create service.yaml for Kubernetes service resource

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.appName }}-entrypoint
  namespace: {{ .Values.namespace }}
spec:
  selector:
    app: {{ .Values.appName }}
  ports:
    - protocol: TCP
      port: 80
      targetPort: {{ .Values.containerPort }}
      name: http

5. Create a generic deployment.yaml for all Springboot services

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.appName }}
  namespace: {{ .Values.namespace }}
spec:
  selector:
    matchLabels:
      app: {{ .Values.appName }}
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ .Values.appName }}
    spec:
      containers:
      - name: {{ .Values.appName }}
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        {{- if .Values.hasSecretVolume }}
        volumeMounts:
        - name: {{ .Values.appName }}-volume-sec
          mountPath: {{ .Values.secretVolumeMountPath }}
        {{- end}}
        {{- if or .Values.env.configMap .Values.env.secrets }}
        envFrom:
        {{- if .Values.env.configMap }}
        - configMapRef:
            name: {{ .Values.appName }}-env-configmap
        {{- end }}
        {{- if .Values.env.secrets }}
        - secretRef:
            name: {{ .Values.appName }}-env-secret
        {{- end }}
        {{- end }}
        ports:
        - containerPort: {{ .Values.containerPort }}
          protocol: TCP
{{- if .Values.springContainerHealthChecks}}
{{ toYaml .Values.springContainerHealthChecks | indent 8 }}
{{- end}}
      {{- if .Values.hasSecretVolume }}
      volumes:
      - name: {{ .Values.appName }}-volume-sec
        secret:
          secretName: {{ .Values.appName }}-volume-sec
      {{- end}}
      {{- if .Values.imageCredentials}}
      imagePullSecrets:
      - name: {{.Values.imageCredentials.secretName}}
      {{- end}}

6. Create an ingress.yaml for Kubernetes ingress resource

{{- if .Values.ingress.enabled }}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ .Values.ingress.name }}
  namespace: {{ .Values.namespace }}
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/rewrite-target: /
    #nginx.org/server-snippet: "proxy_ssl_verify off;"
spec:
  rules:
{{ toYaml .Values.ingress.rules | indent 2 }}
{{- end}}

7. Create _imagepullsecret_helper.tpl template helper to pull docker images

/* image pull secret */
{{- define "imagePullSecret" }}
{{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry (printf "%s:%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }}
{{- end }}

8. Create a namespace/<ENV>/<application_name>/values.yaml to pass values to above created files

# Default values for auth.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Generic properties
namespace: qa
replicaCount: 1
ingressEnabled: true
appName: my-boot-service
containerPort: 8080
hasSecretVolume: true
secretVolumeMountPath: /opt/appConfig

# Images
image:
  repository: my-first-boot/auth
  tag: v1
  pullPolicy: Always
  pullSecrets: regcred

# Image creds
imageCredentials:
  secretName: regcred
  registry: hub.docker.com
  username: abcd
  password: password
  docker-email: chirag.naik@briozing.com
# ----------------------------------------

env:
  configMap:
    JAVA_OPTS: -Dspring.config.location=/opt/appConfig/application.properties -Dspring.profiles.active=qa -Dspring.hikaricp.config.location=/opt/appConfig/
  secrets:
    APPLICATION_NAME: my-boot-service

# Container liveness and readyness
springContainerHealthChecks:
  livenessProbe:
    httpGet:
      path: /api/ping
      port: 8080
    initialDelaySeconds: 180
    timeoutSeconds: 1
    periodSeconds: 15
  readinessProbe:
    httpGet:
      path: /api/ping
      port: 8080
    initialDelaySeconds: 180
    timeoutSeconds: 1
    periodSeconds: 15

# Nginx ingress settings
controller:
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
  service: 
    enableHttp: true
  stats:
    enabled: true
  metrics:
    enabled: true

# Ingress settigns
ingress:
  enabled: true
  name: my-boot-service
  rules:
    - host: my-boot-service
      http:
        paths:
        - path: /api
          backend:
            serviceName: my-boot-service-entrypoint
            servicePort: 80

We did similar steps for Ruby on Rails and Node service and it just varied at deployment.yaml file as their runtime environment differs from each other.

Tags: , , , , ,

Leave a Reply

Your e-mail address will not be published. Required fields are marked *