Using Kubernetes Downscaler for Time-of-Day & Day-of-Week Automation
Note: This is the latest post from our R&D team in our technology blog series. To view the rest of our technology series, please click here.
I am a DevOps Engineer on OpenGov’s EngOps team which manages cloud technologies at scale running in production. These technologies power mission-critical government applications that enable a city, a county, or even a state to function in a remote and distributed manner. If you are interested in more details about our R&D team structure, you can check this blog. To learn more about OpenGov’s high velocity, cutting-edge tech stack from this blog. The impact of the company on our nation’s governments and communities is really impressive, and I am happy to be part of this evolution.
With massive migration of apps and services to the cloud, companies have started to realize savings in costs and time, but there is still room to save more by enabling elastic infrastructure which should be available on-demand or as required. It allows us not to be over or under-provisioned when we have low or peak traffic, finding an optimal balance between consumption and capacity. At OpenGov we have taken big steps in this direction using specific tools and technologies that help us operate an elastic infrastructure for our developers and apps. In this blog, I am going to present a closer look at one such open-source project, the Kubernetes Downscaler.
The need was triggered by one of our applications running in both production and sandbox environments. We observed that this application was consuming a significant amount of our cloud resources. Since the sandbox instance is only intermittently used by customers to verify changes in general before pushing to production, we wanted to reduce the resource usage without impacting our customer experience. The target state was to run sandbox instances during customer’s business hours only, as they normally do not use these applications outside of those hours. We did research to take a closer look at the following options:
- Jenkins
Pros: Seemed straightforward since we used it already.
Cons: it would create a dependency on Jenkins.
- Kubernetes Cron Jobs
Pros: Native Kubernetes solution.
Cons: Had no timezone concept, needed to orchestrate time zones, requiring significant development effort.
- K8S Custom Resource Definition
Pros: Requires us to create our own object kinds and lets the API Server handle it without having another dependency.
Cons: The need to maintain our own solution did not seem exciting in the long-term.
- Kubernetes Downscaler
Pros: Looked perfect for what we needed, by only using annotation to scale the application up or down according to a time zone.
Cons: Still required some level of manual work during breaks and holidays.
After reviewing each option we decided to go with Kubernetes Downscaler which required fewer changes in application and infrastructure side, same time ease of maintenance. Shortly after the execution plan was confirmed by the engineering team, we rolled it to integration and production environments.
We have deployed it in production for around 2 years and can say it’s a piece of software that you can deploy and forget about maintaining. Eventually, it would be nice to have this functionality built-in Kubernetes as part of Horizontal Pod Autoscaler which automatically scales the number of pods in a replication controller, deployment, replica set, or stateful set based on utilization or custom metrics.
Kubernetes Downscaler is written in Python by Henning Jacobs and does an amazing job reducing unnecessary workloads running in Kubernetes clusters. It is a small single pod application deployed in the cluster and instruments resources such as deployments, statefulsets, stacks, namespaces. The application can be configured in multiple ways. The precedence order is as: annotation (deployment, namespace), command-line interface, environment variables. In case you are new to the Kubernetes world, the annotations are the metadata for Kubernetes resources; you can read about them here.
Below are some interesting use cases, it supports:
- Scale down only in a given period of time, as example coming holiday:
DOWNSCALE_PERIOD=”Mon-Mon 00:00-24:00 Europe/Berlin”
- Scale-up only in a given period of time, for example, company event, workshop, hackathon:
UPSCALE_PERIOD=”Sat-Sun 00:00-24:00 Europe/Berlin”
- Regular scale down, for example during weekends, when nobody is at the office, bring down pre-production infrastructure:
DEFAULT_DOWNTIME=”Sat-Sun 00:00-24:00 CET,Fri-Fri 20:00-24:00 CET”
- Regular scale-up, for example during work hours, keep infrastructure running:
DEFAULT_UPTIME=”Mon-Fri 07:30-20:30 Europe/Berlin”
Going to more granularity, the annotations can be set in deployments, stateful sets, stacks, or in namespace level, can include or exclude these resources, set replica count during downscale, run downscaler only once or periodically etc.
In conclusion, using a Kubernetes downscaler we managed to save more than half of our cloud infrastructure cost, while not impacting the customer experience. We also shared this approach with other teams, to encourage the use of it across the company. We strongly believe it is a great value add to our mission to help power a more effective and accountable government.
For more posts from our R&D team blog series, please click here.
Acknowledgments
Thanks to our entire EngOps team for collaborating during project implementation from requirements elicitation till finding a way to production, including our former colleague Jonathan Hinds who was with us throughout the journey.
Special thanks to Henning Jacobs for creating and maintaining the Kubernetes Downscaler project.
If you would like to read more from OpenGov’s engineering team, read the rest of our Technology blog posts.
Interested in contributing to OpenGov’s Engineering culture of innovation, leading-edge technology adoption, and quality? Check out our current job openings.
Category: Technology