# Kubernetes fail compilation: but they keep getting worse I’ve never been a big gambler. I might have placed a small bet in the past to spice up a Super Bowl that I wasn't that invested in, but nothing crazy. There is a level of certainty that's required to actually put your money on the line that I rarely have for any sporting event, electoral outcome, or future prediction. There are very few certainties in tech. Job security isn’t a given, industry trends ebb and flow, and the [tools](https://glasskube.dev/guides/kubectl/) and tech stack you work on every day will more than likely evolve as time goes on. Despite this sea of uncertainty, there is something you can safely bet your money on, at some point, **you will suffer an outage.** ![casino](https://res.cloudinary.com/practicaldev/image/fetch/s--PVj-ttLi--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://media1.giphy.com/media/26tneF8wxg0H4NrC8/200.webp%3Fcid%3Decf05e4725b5vpix4vj3x0ajz06gb8rehvx96kjva729y9nu%26ep%3Dv1_gifs_search%26rid%3D200.webp%26ct%3Dg align="center") Kubernetes engineers who have been around for any amount of time can attest to this reality. This being the case, it makes little sense to fear failure or perform herculean feats to ensure [100% availability](https://andrewmatveychuk.com/why-99-99-uptime-or-sla-is-bad/), if anything [mistakes](https://www.amazon.com/Black-Box-Thinking-People-Mistakes-But/dp/1591848229) and outages should be welcomed as learning opportunities, a necessary evil to any environment that aspires to mature and deliver a high-quality service in a reliable way. The best way to effectively process and digest an outage is the systematic performance of a [post-mortem](https://www.atlassian.com/incident-management/postmortem/reports). These prove to be the sharpest tools we have to find patterns and synthesize the learnings that an outage has to offer. On the topic of patterns, common in Kubernetes cluster failures there are a few that emerge. DNS, networking, and default resource allocation are some of the key culprits. In this article, we will analyze some of these post-mortems and try our best to absorb what others have had to learn the hard way. ## Failure severity scale Not every outage has the same impact, so I’ve created a very scientific categorization system to understand the impact of each Kubernetes outage: **🤷 - Oopsie Daisy:** Trivial Kubernetes failures **😅 - Non-prod, but still annoying:** no customers we affected but learnings were made **🤬 - Houston, we have a problem in production:** where customers were impacted and career choices were questioned. ![you're fired](https://res.cloudinary.com/practicaldev/image/fetch/s---T3bQ9Fy--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://media1.giphy.com/media/xT4uQ7N8UNsoeFAjVS/200.webp%3Fcid%3D790b76115lk6lg7hzua5e4qexoocj2f0sltdsw5l4ggfcsln%26ep%3Dv1_gifs_search%26rid%3D200.webp%26ct%3Dg align="center") --- Before I forget, let me thank [Glasskube](https://github.com/glasskube/glasskube) for allowing me to take the time to create content just like this. If this is the first time you've heard of us, we are working to build the next generation `Package Manager for Kubernetes`. If you'd like to support us on this mission we would appreciate it if you could [⭐️ Star Glasskube on GitHub 🙏](https://github.com/glasskube/glasskube) ## Oopsie Daisy 🤷 If you can’t laugh at yourself who can you laugh at? ### Clusters and node groups The first story comes from your humble correspondent, who recently spun up a test Kubernetes cluster using the AWS console for a quick proof of concept. It had been a while since I'd created a cluster without using EKSCTL or some form of Infrastructure as Code definition file. So, I logged in, accessed the EKS console, named my Kubernetes cluster, and hit "create." I then followed the CLI instructions to configure the kubeconfig file and connect to my newly created cluster via the terminal. Eager to test the newest version of Glasskube, I installed it in the cluster. However, I was surprised by how long the pods were taking to schedule. Reflecting on it now, I’m embarrassed to admit how long it took me to realize that I hadn’t provisioned a node group, no wonder the pods weren’t being scheduled. ![unimpressed](https://res.cloudinary.com/practicaldev/image/fetch/s--iXeplLTR--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://media4.giphy.com/media/v1.Y2lkPTc5MGI3NjExNm9mdzVweWc1cDljdjM3MWpmeTRqcm1sejltZXhzZmZlYzFtMmNtMSZlcD12MV9naWZzX3NlYXJjaCZjdD1n/c5FhF1waAJ5wk/giphy.webp align="center") ### Call the fire brigade, I forgot to add resource limits. Another true story comes from another Glasskube member who overloaded his local laptop due to installing to many components (GitLab) in his local Minikube cluster, the laptop nearly burned a hole through his desk, a good reminder to use resource limits and requests ![laptop fire](https://res.cloudinary.com/practicaldev/image/fetch/s--r66EBK8h--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://media0.giphy.com/media/dbtDDSvWErdf2/200.webp%3Fcid%3Decf05e4775sy7d66ores8c10w1hrw77311mm7m323lkkkqym%26ep%3Dv1_gifs_search%26rid%3D200.webp%26ct%3Dg align="center") ## Non-prod, but still annoying 😅 Moving on to some real incidents, these luckily were localized to clusters that didn’t impact paying customers. ### Incident #1: Venafi’s unresponsive Webhooks [Venafi](https://venafi.com/) is a Control Plane for Machine Identities, recently acquired by [Cyberark](https://www.cyberark.com/) who had some issues with OPA. Full post-mortem [here](https://venafi.com/blog/gke-webhook-outage/). > **Impact:** intermittent API server timeouts leading to unhealthy nodes. > **Involved:** Open Policy Agent, Node readiness **Play-by-play** During a scheduled cluster upgrade and despite warnings and successful prior upgrades, the master upgrade failed, leading to API server timeouts and node instability. **The root cause was a timeout during a ConfigMap update**, triggered by an unresponsive [OPA](https://www.openpolicyagent.org/) webhook. Deleting the webhook restored service, and they've since restricted it to specific namespaces, added a liveness probe for OPA, and updated documentation. They emphasized the need for API response time alerts, workload probes, and possibly using a Helm chart for deployment to avoid similar issues in the future. They continue to monitor improvements in functionality and offer insights through their Flightdeck service. **Learnings:** * The need for alerting on API server response times. * Increased `livenessProbes` needed for all workloads. * Using package management for more granular configuration. > 💡 This incident highlights one of the use cases [Glasskube](https://github.com/glasskube/glasskube) aims to address. While Glasskube doesn't yet support the OPA operator, we believe this issue could have been avoided with a robust Kubernetes package manager. Glasskube allows for easy configuration of key features, assists in upgrades, and applies a GitOps approach to package operator management, including rollbacks and specific namespace allocation. Try it [here](https://github.com/glasskube/glasskube). ### Incident #2: When crypto miners sneak in [JW Player](https://jwplayer.com/) was targeted by bitcoin mining malware, check out the full post-mortem [here](https://medium.com/jw-player-engineering/how-a-cryptocurrency-miner-made-its-way-onto-our-internal-kubernetes-clusters-9b09c4704205). > **Impact:** A non-prod cluster was infiltrated by bitcoin miners > **Involved:** Root access exploitation **Play-by-play** The DevOps team at JW Player discovered a cryptocurrency miner on their Kubernetes clusters after Datadog alerted them to high load averages in their staging and development environments. **Initial investigation pointed to a gcc process consuming 100% CPU, which was found to be a miner** launched by Weave Scope, a monitoring tool. The miner exploited a public-facing Weave Scope load balancer that allowed command execution in containers. Immediate actions included stopping Weave Scope, isolating affected nodes, and rotating them out. The incident caused high CPU usage but no service disruption or data compromise. The team identified manual security group edits overridden by Kubernetes as a key issue and emphasized the need for proper configuration practices to prevent such vulnerabilities. **Learnings:** * Monitoring load is not the best way to detect cluster issues. * Tools like `falcon` or `sysdig` might be needed * More robust Docker image and container scanning needed. * Some areas of the architecture need revisiting. * More cross-team data sharing and communication is needed. ### Incident #3: GKE ran out of IP addresses > **Impact:** A high node-count cluster ran out of IP addresses and couldn’t schedule new pods. > **Involved:** Subnets, Default IP allocations per node **Play-by-play** An incident arose when a team member reported unusually long deployment times for their application. They discovered quickly that while some newly deployed pods were serving traffic, the rest remained in a `pending` state. **Analysis revealed a** `FailedScheduling` warning indicating insufficient resources. Despite having a cluster autoscaler in place, the issue persisted, as they saw an alarming **"0/256 nodes available"** message. Further examination uncovered that GKE pre-allocates 110 IPs per node, resulting in unexpected high IP consumption. Once this was known, they adjusted the pod allocation per node, reducing overall IP usage by 30%. Additionally, they explored options like subnet expansion and increasing node sizes to mitigate IP exhaustion, eventually optimizing node pool instance sizes to better utilize resources. **Learnings:** * The importance of knowing the default values set by GKE. * [Subnet expansion](https://cloud.google.com/vpc/docs/create-modify-vpc-networks#expand-subnet) is a nifty tool to have at your disposal (not much documentation on secondary ranges though). * Increased node pool instance size can do the job too (running more pods per node, then needing fewer nodes). ## Houston, we have a problem in production 🤬 These are the types of outages that **keep SRE's up at night**, when customers are impacted and business value is on the line, these are where the most important learnings emerge and where hero's are made. ### Incident #1 Skyscanner only needed a couple of characters to bring their site down Here we see that an architecture optimized for resiliency was still susceptible to failure due to just one line of code. Full post-mortem [here](https://medium.com/@SkyscannerEng/how-a-couple-of-characters-brought-down-our-site-356ccaf1fbc3). > **Impact:** Global Skyscanner website and mobile apps were inaccessible > **Involved:** IaC manifest **Play-by-play** In August 2021, Skyscanner faced a global outage lasting over four hours due to an inadvertent change to a root file in its infrastructure provisioning system. **This change, with a lack of** `{{ }}`, unexpectedly triggered the deletion of critical microservices across the globe, rendering the website and mobile apps inaccessible. They swiftly addressed the issue, leveraging GitOps to restore configurations and prioritize critical services. **Learnings:** * Don’t do global config deploys. * More drastic “worst case scenario“ planning is needed. * Verify the backup/restore process. * Keep runbooks up-to-date. * Potential over automation. ### Incident #2 Monzo Bank’s linkerd fiasco British digital [bank](https://monzo.com/) found a critical Kubernetes bug the hard way. Full port-mortem [here](https://community.monzo.com/t/resolved-current-account-payments-may-fail-major-outage-27-10-2017/26296/95). > **Impact:** prepaid cards and new current accounts were down for about 1.5 hours > **Involved:** Linkerd, kube-apiserver, etcd **Play-by-play:** The incident began when a routine deployment caused payment processing failures. Attempts to rollback the change were unsuccessful, leading to an internal outage declaration. Engineers identified and restarted unhealthy `linkerd` instances, but a configuration issue with `kube-apiserver` prevented new `linkerd` instances from starting, escalating the outage to a full platform failure. **The root cause was traced to a bug in Kubernetes and etcd, triggered by a recent cluster reconfiguration.** This caused `linkerd` to fail to receive network updates, compounded by a compatibility issue between Kubernetes and linkerd. The incident was resolved by updating linkerd and removing empty Kubernetes services. **Learnings:** * A new version of Linkerd was needed. * The k8s bug needed to be fixed (now [fixed](https://github.com/kubernetes/kubernetes/issues/47131)). * Improve health checks, dashboard, and alerting. * Procedural improvements to improve internal communication during outages. ### Incident #3 The Redis operator threw a curve ball Palark is a DevOps service provider that tried to protect it's Redis cluster and ended up rueing the day. [Here](https://blog.palark.com/failure-with-redis-operator-and-redis-data-analysis-tools/) is the full port-mortem > **Impact:** Production Redis data after adding replicas > **Involved:** Redis operator **Play-by-play** They encountered an incident involving the well-known in-memory key-value store, Redis, which they installed via the [Redis Operator](https://github.com/spotahome/redis-operator) for running Redis failover. Initially deployed with one Redis replica, they expanded to two replicas to enhance database reliability. **However, this seemingly minor change proved catastrophic during a rollout, leading to data loss.** The incident exposed flaws in the Redis Operator, primarily its `readiness probe`, triggering unintended master promotion and subsequent data destruction. Further analysis using tools like `Redis-memory-analyzer` revealed insights into database size and elements which then helped developers to optimise the database and application code to prevent future incidents. **Learnings:** * To be very careful when using Kubernetes operators (make sure they are mature and well-tested). * The found a crucial bug associated the the Redis Operators readiness probe that made replica scale out prone to data loss (since [fixed](https://github.com/spotahome/redis-operator/releases/tag/v1.0.0-rc.3)). * `Redis-memory-analyzer` is the best tool for troubleshooting Redis databases. ## Incident #4 Datadog's, multi-region nightmare Multiple [Datadog](https://www.datadoghq.com/) regions were down `systemd-networkd` forcibly deleted the routes managed by the Container Network Interface (CNI) plugin. Full post-mortem [here](https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/). > **Impact:** Users in multiple regions were left without API and platform access. > **Involved:** systemd update, Cilium **Play-by-play** Starting on March 8, 2023, Datadog experienced a major outage affecting multiple regions, preventing users from accessing the platform, APIs, and monitors, and impacting data ingestion. The issue, triggered by an automatic security update to systemd on numerous VMs, caused network disruptions that took tens of thousands of nodes offline. Recovery involved restoring compute capacity, addressing service-specific issues, and providing continuous updates to customers. **The root cause was identified as a misconfiguration that allowed the automatic update, which has since been disabled.** **Learnings:** * More robust chaos testing. * Improved communication with customers during outages is needed * The status page was inadequate during the outage. * Automatic updates are inherently risky and should be employed with care. ### Incident #5 Reddit’s Pi-Day Outage Reddit suffered the consequences of rapid organic growth, they faced the crushing reality that a lot of their critical Kubernetes clusters were unstandardised and susceptible to outages, full Pi-Day outage post-mortem [here](https://www.reddit.com/r/RedditEng/comments/11xx5o0/you_broke_reddit_the_piday_outage/). > **Impact:** Significant cross-platform outage lasting 314 minutes > **Involved:** Calico, Kubernetes version update **Play-by-play** In March 2023, Reddit experienced a significant outage lasting 314 minutes, coincidentally occurring on Pi Day. Users trying to access the site encountered either an overwhelmed Snoo mascot, error messages, or an empty homepage. This outage was triggered by an upgrade from Kubernetes 1.23 to 1.24, which introduced a subtle, previously unseen issue. The engineering team, having emphasized improvements in availability over recent years, found themselves in a challenging situation where a rollback, though risky, became the best option. During the restore, complications arose from mismatches in TLS certificates and AWS capacity limits, but the team managed to navigate these challenges and reestablish a high-availability control plane. **Further investigation revealed that the root cause was related to an outdated route reflector configuration for Calico**, which became incompatible with Kubernetes 1.24 due to the removal of the "master" node label. **Learnings:** * The importance of improving the pre-prod cluster for testing purposes. * The need for improved Kubernetes component lifecycle management tooling. * Need for more homogeneous environments. * Also, the need to increase their IaC and internal technical documentation. ## Conclusion As you can see, the law of entropy easily applies to Kubernetes clusters—it's much easier to break them than to keep them happy. Changes like upgrades, rollouts, scale-outs, and deployments usually trigger outages, so you might feel inclined to minimize them. But this isn’t an option for organizations fighting to lead their market segments and meet changing customer needs. The best we can hope for is to learn by doing and learn by failing. On the upside, the tech industry is generally open to learning and upfront about failures ([for the most part](https://apnews.com/article/cellular-att-verizon-tmobile-outage-02d8dfd93019e79e5e2edbeed08ee450)). The fact that many large enterprises publicly share post-mortem summaries for the greater community to learn from is a best practice grounded in the assumption that failures and outages are a matter of “when” and not “if.” The best way to protect ourselves is to learn from them once they have passed. > 🫵 And what about you? Have you weathered any particularly difficult outages and come out the other side to tell the tale? If so, please share your experience in the comments below. I'm sure many of us would love to hear about it. --- ## Help us make more content like this! At [Glasskube](https://github.com/glasskube/glasskube) we're putting a lot of effort into content just like this, as well as building the `next generation package manager for Kubernetes`. If you get value from the work we do, we'd appreciate it if you could [⭐️ Star Glasskube on GitHub 🙏](https://github.com/glasskube/glasskube)