Platform Engineering: How to build green Kubernetes Platform for your business ?
Note: This is hand written blog . In this blog I will walk through the guide that help to reduce a lots of pain in your infra setup with Kubernetes.
Let’s understand the problem statement at first. So, development team come up with the bunch of application repo and your task is to deploy that application to Kubernetes but it should be very automated and production ready along with Observability setup. In this blog we gonna discuss about tooling and approach that we will use to build our platform on the top of the Kubernetes. Let’s go through it step by step ….
- Containerization
Start with Dockerfile. If developer don’t have :)
2. Setup Up CI for each application
Once, we have Dockerfile, we will create the CI pipeline that build, in between all the stuff depending on your requirement and push the image to any private Repo.
3. Setup the Infra Repo
Infra repo is going to have all the k8s manifest and terraform code. This is going to be a single source of truth for your entire setup. The folder structure will look like:
apps folder going to have your core application which will be created using Kustomize pattern base, dev, stage, prod e.t.c depending on your number of environment.
argocd-apps folder going to have all the argocd related application for your different environment for eg: argocd-apps/dev, argocd-apps/stage, argocd-apps/prod. There will be another folder which is argocd-app/apps-of-app that contains single app that going to deploy all the argocd application for each environment.
platform folder is going to contain all the application same in Kustomize pattern. The heart of your system will go here. Let’s elaborate this in below section
terraform folder is going to have code for spinning the infra for different environment written in modular format and this is will executed via CI.
4. Things that needs to be installed on top of kubernetes
This is the very important for CD part which take our infra repo as single source of truth and maintain the git state and cluster state same.
Once, CI pushes a new Image, there should be something that automatically update the new image to the cluster. In this case, I will highly recommend using ArgoImageUpdater.
Store the secret data at private key store and pull the value for corresponding key and create the target secret with that key/value. This operator makes this entire activity possible.
If you don’t want to restart your pod manually when there will be change in configmap and secret then use Reloader for that.
For Certificate management use cert-manager.
Ingress is the old school way, I would recommend to shift to Gateway API.
OpenTelemetry is the first thing that I always install at first in the cluster while doing observalibilty setup. Auto-instrumentation is another pretty dope things that you need to explore. Once, we have logs, metrics and traces from Auto-instrumentation. I will recommend to use LGTM (Loki, Grafana, Tempo, Mimir) stack for full stack Observability.
I remember that one of my friend deleted the whole namespace from dev environment which causes the data loss of 3 months. If you want to setup setup of rule at admission level and restrict such incident then I would recommend to use kyverno. We also have OPA as well.
Use velero to take cluster backup.
If you are struggling with cost and CloudSQL is costing alot then switch to Cloud NativePG Operator.
Cost has been always a painful part for every Organization. We have kubecost, OpenCost for cost management in Kubernetes that you can take a look.
There is another enterprise product called Cast AI for cost management that I have been using for some time and results are really impressive. We have saved thousands of dollar.
Note: I am not a advocate of Cast AI, above statement are based on my experience.
Please don’t blame technology. I see a lot of people complaining that Kubernetes is not right for this or that business. If you know that Kubernetes is going to burn you out, then run your application on a VM. If it works for you, who cares? The problem is not with the technology; it’s with some decision-maker inside your organization.
Using the above setup, tools, and approach, I am sure your system will run seamlessly. There will be a lot of tooling that increases based on your requirements. The above tools are sort of the best and mandatory ones.
If you think something is missing, please put it in comment section. If you like it make sure to clap and share :)