# Gateway API v1.1: Service mesh, GRPCRoute, and a whole lot more By [**Richard Belleville**](https://github.com/gnossen) **(Google),** [**Frank Budinsky**](https://github.com/frankbu) **(IBM),** [**Arko Dasgupta**](https://github.com/arkodg) **(Tetrate),** [**Flynn**](https://github.com/kflynn) **(Buoyant),** [**Candace Holman**](https://github.com/candita) **(Red Hat),** [**John Howard**](https://github.com/howardjohn) **(Solo.io),** [**Christine Kim**](https://github.com/xtineskim) **(Isovalent),** [**Mattia Lavacca**](https://github.com/mlavacca) **(Kong),** [**Keith Mattix**](https://github.com/keithmattix) **(Microsoft),** [**Mike Morris**](https://github.com/mikemorris) **(Microsoft),** [**Rob Scott**](https://github.com/robscott) **(Google),** [**Grant Spence**](https://github.com/gcs278) **(Red Hat),** [**Shane Utt**](https://github.com/shaneutt) **(Kong),** [**Gina Yeh**](https://github.com/ginayeh) **(Google), and other review and release note contributors** | Thursday, May 09, 2024 ![Gateway API logo](https://kubernetes.io/blog/2024/05/09/gateway-api-v1-1/gateway-api-logo.svg align="left") Following the GA release of Gateway API last October, Kubernetes SIG Network is pleased to announce the v1.1 release of [Gateway API](https://gateway-api.sigs.k8s.io/). In this release, several features are graduating to *Standard Channel* (GA), notably including support for service mesh and GRPCRoute. We're also introducing some new experimental features, including session persistence and client certificate verification. ## **What's new** ### **Graduation to Standard** This release includes the graduation to Standard of four eagerly awaited features. This means they are no longer experimental concepts; inclusion in the Standard release channel denotes a high level of confidence in the API surface and provides guarantees of backward compatibility. Of course, as with any other Kubernetes API, Standard Channel features can continue to evolve with backward-compatible additions over time, and we certainly expect further refinements and improvements to these new features in the future. For more information on how all of this works, refer to the [Gateway API Versioning Policy](https://gateway-api.sigs.k8s.io/concepts/versioning/). #### [**Service Mesh Support**](https://gateway-api.sigs.k8s.io/mesh/) Service mesh support in Gateway API allows service mesh users to use the same API to manage ingress traffic and mesh traffic, reusing the same policy and routing interfaces. In Gateway API v1.1, routes (such as HTTPRoute) can now have a Service as a `parentRef`, to control how traffic to specific services behave. For more information, read the [Gateway API service mesh documentation](https://gateway-api.sigs.k8s.io/mesh/) or see the [list of Gateway API implementations](https://gateway-api.sigs.k8s.io/implementations/#service-mesh-implementation-status). As an example, one could do a canary deployment of a workload deep in an application's call graph with an HTTPRoute as follows: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: color-canary namespace: faces spec: parentRefs: - name: color kind: Service group: "" port: 80 rules: - backendRefs: - name: color port: 80 weight: 50 - name: color2 port: 80 weight: 50 ``` This would split traffic sent to the `color` Service in the `faces` namespace 50/50 between the original `color` Service and the `color2` Service, using a portable configuration that's easy to move from one mesh to another. #### [**GRPCRoute**](https://gateway-api.sigs.k8s.io/guides/grpc-routing/) If you are already using the experimental version of GRPCRoute, we recommend holding off on upgrading to the standard channel version of GRPCRoute until the controllers you're using have been updated to support GRPCRoute v1. Until then, it is safe to upgrade to the experimental channel version of GRPCRoute in v1.1 that includes both v1alpha2 and v1 API versions. #### [**ParentReference Port**](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.ParentReference) The `port` field was added to ParentReference, allowing you to attach resources to Gateway Listeners, Services, or other parent resources (depending on the implementation). Binding to a port also allows you to attach to multiple Listeners at once. For example, you can attach an HTTPRoute to one or more specific Listeners of a Gateway as specified by the Listener `port`, instead of the Listener `name` field. For more information, see [Attaching to Gateways](https://gateway-api.sigs.k8s.io/api-types/httproute/#attaching-to-gateways). #### [**Conformance Profiles and Reports**](https://gateway-api.sigs.k8s.io/concepts/conformance/#conformance-profiles) The conformance report API has been expanded with the `mode` field (intended to specify the working mode of the implementation), and the `gatewayAPIChannel` (standard or experimental). The `gatewayAPIVersion` and `gatewayAPIChannel` are now filled in automatically by the suite machinery, along with a brief description of the testing outcome. The Reports have been reorganized in a more structured way, and the implementations can now add information on how the tests have been run and provide reproduction steps. ### **New additions to Experimental channel** #### [**Gateway Client Certificate Verification**](https://gateway-api.sigs.k8s.io/geps/gep-91/) Gateways can now configure client cert verification for each Gateway Listener by introducing a new `frontendValidation` field within `tls`. This field supports configuring a list of CA Certificates that can be used as a trust anchor to validate the certificates presented by the client. The following example shows how the CACertificate stored in the `foo-example-com-ca-cert` ConfigMap can be used to validate the certificates presented by clients connecting to the `foo-https` Gateway Listener. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: client-validation-basic spec: gatewayClassName: acme-lb listeners: name: foo-https protocol: HTTPS port: 443 hostname: foo.example.com tls: certificateRefs: kind: Secret group: "" name: foo-example-com-cert frontendValidation: caCertificateRefs: kind: ConfigMap group: "" name: foo-example-com-ca-cert ``` #### [**Session Persistence and BackendLBPolicy**](https://gateway-api.sigs.k8s.io/geps/gep-1619/) [Session Persistence](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.SessionPersistence) is being introduced to Gateway API via a new policy ([BackendLBPolicy](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1alpha2.BackendLBPolicy)) for Service-level configuration and as fields within HTTPRoute and GRPCRoute for route-level configuration. The BackendLBPolicy and route-level APIs provide the same session persistence configuration, including session timeouts, session name, session type, and cookie lifetime type. Below is an example configuration of `BackendLBPolicy` that enables cookie-based session persistence for the `foo` service. It sets the session name to `foo-session`, defines absolute and idle timeouts, and configures the cookie to be a session cookie: ```yaml apiVersion: gateway.networking.k8s.io/v1alpha2 kind: BackendLBPolicy metadata: name: lb-policy namespace: foo-ns spec: targetRefs: - group: core kind: service name: foo sessionPersistence: sessionName: foo-session absoluteTimeout: 1h idleTimeout: 30m type: Cookie cookieConfig: lifetimeType: Session ``` ### **Everything else** #### [**TLS Terminology Clarifications**](https://gateway-api.sigs.k8s.io/geps/gep-2907/) As part of a broader goal of making our TLS terminology more consistent throughout the API, we've introduced some breaking changes to BackendTLSPolicy. This has resulted in a new API version (v1alpha3) and will require any existing implementations of this policy to properly handle the version upgrade, e.g. by backing up data and uninstalling the v1alpha2 version before installing this newer version. Any references to v1alpha2 BackendTLSPolicy fields will need to be updated to v1alpha3. Specific changes to fields include: * `targetRef` becomes `targetRefs` to allow a BackendTLSPolicy to attach to multiple targets * `tls` becomes `validation` * `tls.caCertRefs` becomes `validation.caCertificateRefs` * `tls.wellKnownCACerts` becomes `validation.wellKnownCACertificates` For a full list of the changes included in this release, please refer to the [v1.1.0 release notes](https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.1.0). ## **Gateway API background** The idea of Gateway API was initially [proposed](https://youtu.be/Ne9UJL6irXY?si=wgtC9w8PMB5ZHil2) at the 2019 KubeCon San Diego as the next generation of Ingress API. Since then, an incredible community has formed to develop what has likely become the [most collaborative API in Kubernetes history](https://www.youtube.com/watch?v=V3Vu_FWb4l4). Over 200 people have contributed to this API so far, and that number continues to grow. The maintainers would like to thank *everyone* who's contributed to Gateway API, whether in the form of commits to the repo, discussion, ideas, or general support. We literally couldn't have gotten this far without the support of this dedicated and active community. ## **Try it out** Unlike other Kubernetes APIs, you don't need to upgrade to the latest version of Kubernetes to get the latest version of Gateway API. As long as you're running Kubernetes 1.26 or later, you'll be able to get up and running with this version of Gateway API. To try out the API, follow our [Getting Started Guide](https://gateway-api.sigs.k8s.io/guides/). ## **Get involved** There are lots of opportunities to get involved and help define the future of Kubernetes routing APIs for both ingress and service mesh. * Check out the [user guides](https://gateway-api.sigs.k8s.io/guides) to see what use-cases can be addressed. * Try out one of the [existing Gateway controllers](https://gateway-api.sigs.k8s.io/implementations/). * Or [join us in the community](https://gateway-api.sigs.k8s.io/contributing/) and help us build the future of Gateway API together! ## **Related Kubernetes blog articles** * [New Experimental Features in Gateway API v1.0](https://kubernetes.io/blog/2023/11/28/gateway-api-ga/) 11/2023 * [Gateway API v1.0: GA Release](https://kubernetes.io/blog/2023/10/31/gateway-api-ga/) 10/2023 * [Introducing ingress2gateway; Simplifying Upgrades to Gateway API](https://kubernetes.io/blog/2023/10/25/introducing-ingress2gateway/) 10/2023 * [Gateway API v0.8.0: Introducing Service Mesh Support](https://kubernetes.io/blog/2023/08/29/gateway-api-v0-8/) 08/2023 # **Completing the largest migration in Kubernetes history** By **Andrew Sy Kim (Google), Michelle Au (Google), Walter Fender (Google), Michael McCune (Red Hat)** | Monday, May 20, 2024 Since as early as Kubernetes v1.7, the Kubernetes project has pursued the ambitious goal of removing built-in cloud provider integrations ([KEP-2395](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/2395-removing-in-tree-cloud-providers/README.md)). While these integrations were instrumental in Kubernetes' early development and growth, their removal was driven by two key factors: the growing complexity of maintaining native support for every cloud provider across millions of lines of Go code, and the desire to establish Kubernetes as a truly vendor-neutral platform. After many releases, we're thrilled to announce that all cloud provider integrations have been successfully migrated from the core Kubernetes repository to external plugins. In addition to achieving our initial objectives, we've also significantly streamlined Kubernetes by removing roughly 1.5 million lines of code and reducing the binary sizes of core components by approximately 40%. This migration was a complex and long-running effort due to the numerous impacted components and the critical code paths that relied on the built-in integrations for the five initial cloud providers: Google Cloud, AWS, Azure, OpenStack, and vSphere. To successfully complete this migration, we had to build four new subsystems from the ground up: 1. **Cloud controller manager** ([KEP-2392](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/2392-cloud-controller-manager/README.md)) 2. **API server network proxy** ([KEP-1281](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1281-network-proxy)) 3. **kubelet credential provider plugins** ([KEP-2133](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2133-kubelet-credential-providers)) 4. **Storage migration to use** [**CSI**](https://github.com/container-storage-interface/spec?tab=readme-ov-file#container-storage-interface-csi-specification-) ([KEP-625](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/625-csi-migration/README.md)) Each subsystem was critical to achieve full feature parity with built-in capabilities and required several releases to bring each subsystem to GA-level maturity with a safe and reliable migration path. More on each subsystem below. ### **Cloud controller manager** The cloud controller manager was the first external component introduced in this effort, replacing functionality within the kube-controller-manager and kubelet that directly interacted with cloud APIs. This essential component is responsible for initializing nodes by applying metadata labels that indicate the cloud region and zone a Node is running on, as well as IP addresses that are only known to the cloud provider. Additionally, it runs the service controller, which is responsible for provisioning cloud load balancers for Services of type LoadBalancer. ![Kubernetes components](https://kubernetes.io/images/docs/components-of-kubernetes.svg align="left") To learn more, read [Cloud Controller Manager](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) in the Kubernetes documentation. ### **API server network proxy** The API Server Network Proxy project, initiated in 2018 in collaboration with SIG API Machinery, aimed to replace the SSH tunneler functionality within the kube-apiserver. This tunneler had been used to securely proxy traffic between the Kubernetes control plane and nodes, but it heavily relied on provider-specific implementation details embedded in the kube-apiserver to establish these SSH tunnels. Now, the API Server Network Proxy is a GA-level extension point within the kube-apiserver. It offers a generic proxying mechanism that can route traffic from the API server to nodes through a secure proxy, eliminating the need for the API server to have any knowledge of the specific cloud provider it is running on. This project also introduced the Konnectivity project, which has seen growing adoption in production environments. You can learn more about the API Server Network Proxy from its [README](https://github.com/kubernetes-sigs/apiserver-network-proxy#readme). ### **Credential provider plugins for the kubelet** The Kubelet credential provider plugin was developed to replace the kubelet's built-in functionality for dynamically fetching credentials for image registries hosted on Google Cloud, AWS, or Azure. The legacy capability was convenient as it allowed the kubelet to seamlessly retrieve short-lived tokens for pulling images from GCR, ECR, or ACR. However, like other areas of Kubernetes, supporting this required the kubelet to have specific knowledge of different cloud environments and APIs. Introduced in 2019, the credential provider plugin mechanism offers a generic extension point for the kubelet to execute plugin binaries that dynamically provide credentials for images hosted on various clouds. This extensibility expands the kubelet's capabilities to fetch short-lived tokens beyond the initial three cloud providers. To learn more, read [kubelet credential provider for authenticated image pulls](https://kubernetes.io/docs/concepts/containers/images/#kubelet-credential-provider). ### **Storage plugin migration from in-tree to CSI** The Container Storage Interface (CSI) is a control plane standard for managing block and file storage systems in Kubernetes and other container orchestrators that went GA in 1.13. It was designed to replace the in-tree volume plugins built directly into Kubernetes with drivers that can run as Pods within the Kubernetes cluster. These drivers communicate with kube-controller-manager storage controllers via the Kubernetes API, and with kubelet through a local gRPC endpoint. Now there are over 100 CSI drivers available across all major cloud and storage vendors, making stateful workloads in Kubernetes a reality. However, a major challenge remained on how to handle all the existing users of in-tree volume APIs. To retain API backwards compatibility, we built an API translation layer into our controllers that will convert the in-tree volume API into the equivalent CSI API. This allowed us to redirect all storage operations to the CSI driver, paving the way for us to remove the code for the built-in volume plugins without removing the API. You can learn more about In-tree Storage migration in [Kubernetes In-Tree to CSI Volume Migration Moves to Beta](https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/). ## **What's next?** This migration has been the primary focus for SIG Cloud Provider over the past few years. With this significant milestone achieved, we will be shifting our efforts towards exploring new and innovative ways for Kubernetes to better integrate with cloud providers, leveraging the external subsystems we've built over the years. This includes making Kubernetes smarter in hybrid environments where nodes in the cluster can run on both public and private clouds, as well as providing better tools and frameworks for developers of external providers to simplify and streamline their integration efforts. With all the new features, tools, and frameworks being planned, SIG Cloud Provider is not forgetting about the other side of the equation: testing. Another area of focus for the SIG's future activities is the improvement of cloud controller testing to include more providers. The ultimate goal of this effort being to create a testing framework that will include as many providers as possible so that we give the Kubernetes community the highest levels of confidence about their Kubernetes environments. If you're using a version of Kubernetes older than v1.29 and haven't migrated to an external cloud provider yet, we recommend checking out our previous blog post [Kubernetes 1.29: Cloud Provider Integrations Are Now Separate Components](https://kubernetes.io/blog/2023/12/14/cloud-provider-integration-changes/).It provides detailed information on the changes we've made and offers guidance on how to migrate to an external provider. Starting in v1.31, in-tree cloud providers will be permanently disabled and removed from core Kubernetes components. If you’re interested in contributing, come join our [bi-weekly SIG meetings](https://github.com/kubernetes/community/tree/master/sig-cloud-provider#meetings)! # **10 Years of Kubernetes** By [**Bob Killen**](https://github.com/mrbobbytables) **(CNCF),** [**Chris Short**](https://github.com/chris-short) **(AWS),** [**Frederico Muñoz**](https://github.com/fsmunoz) **(SAS),** [**Kaslin Fields**](https://github.com/kaslin) **(Google),** [**Tim Bannister**](https://github.com/sftim) **(The Scale Factory), and every contributor across the globe** | Thursday, June 06, 2024 ![KCSEU 2024 group photo](https://kubernetes.io/blog/2024/06/06/10-years-of-kubernetes/kcseu2024.jpg align="left") Ten (10) years ago, on June 6th, 2014, the [first commit](https://github.com/kubernetes/kubernetes/commit/2c4b3a562ce34cddc3f8218a2c4d11c7310e6d56) of Kubernetes was pushed to GitHub. That first commit with 250 files and 47,501 lines of go, bash and markdown kicked off the project we have today. Who could have predicted that 10 years later, Kubernetes would grow to become one of the largest Open Source projects to date with over [88,000 contributors](https://k8s.devstats.cncf.io/d/24/overall-project-statistics?orgId=1) from more than [8,000 companies](https://www.cncf.io/reports/kubernetes-project-journey-report/), across 44 countries. ![KCSCN 2019](https://kubernetes.io/blog/2024/06/06/10-years-of-kubernetes/kcscn2019.jpg align="left") This milestone isn't just for Kubernetes but for the Cloud Native ecosystem that blossomed from it. There are close to [200 projects](https://all.devstats.cncf.io/d/18/overall-project-statistics-table?orgId=1) within the CNCF itself, with contributions from [240,000+ individual contributors](https://all.devstats.cncf.io/d/18/overall-project-statistics-table?orgId=1) and thousands more in the greater ecosystem. Kubernetes would not be where it is today without them, the [7M+ Developers](https://www.cncf.io/blog/2022/05/18/slashdata-cloud-native-continues-to-grow-with-more-than-7-million-developers-worldwide/), and the even larger user community that have all helped shape the ecosystem that it is today. ## **Kubernetes' beginnings - a converging of technologies** The ideas underlying Kubernetes started well before the first commit, or even the first prototype ([which came about in 2013](https://kubernetes.io/blog/2018/07/20/the-history-of-kubernetes-the-community-behind-it/)). In the early 2000s, Moore's Law was well in effect. Computing hardware was becoming more and more powerful at an incredibly fast rate. Correspondingly, applications were growing more and more complex. This combination of hardware commoditization and application complexity pointed to a need to further abstract software from hardware, and solutions started to emerge. Like many companies at the time, Google was scaling rapidly, and its engineers were interested in the idea of creating a form of isolation in the Linux kernel. Google engineer Rohit Seth described the concept in an [email in 2006](https://lwn.net/Articles/199643/): > We use the term container to indicate a structure against which we track and charge utilization of system resources like memory, tasks, etc. for a Workload. ![The future of Linux containers](https://kubernetes.io/blog/2024/06/06/10-years-of-kubernetes/future.png align="left") In March of 2013, a 5-minute lightning talk called ["The future of Linux Containers," presented by Solomon Hykes at PyCon](https://youtu.be/wW9CAH9nSLs?si=VtK_VFQHymOT7BIB), introduced an upcoming open source tool called "Docker" for creating and using Linux Containers. Docker introduced a level of usability to Linux Containers that made them accessible to more users than ever before, and the popularity of Docker, and thus of Linux Containers, skyrocketed. With Docker making the abstraction of Linux Containers accessible to all, running applications in much more portable and repeatable ways was suddenly possible, but the question of scale remained. Google's Borg system for managing application orchestration at scale had adopted Linux containers as they were developed in the mid-2000s. Since then, the company had also started working on a new version of the system called "Omega." Engineers at Google who were familiar with the Borg and Omega systems saw the popularity of containerization driven by Docker. They recognized not only the need for an open source container orchestration system but its "inevitability," as described by Brendan Burns in [this blog post](https://kubernetes.io/blog/2018/07/20/the-history-of-kubernetes-the-community-behind-it/). That realization in the fall of 2013 inspired a small team to start working on a project that would later become **Kubernetes**. That team included Joe Beda, Brendan Burns, Craig McLuckie, Ville Aikas, Tim Hockin, Dawn Chen, Brian Grant, and Daniel Smith. ## **A decade of Kubernetes** ![KubeCon EU 2017](https://kubernetes.io/blog/2024/06/06/10-years-of-kubernetes/kubeconeu2017.jpg align="left") Kubernetes' history begins with that historic commit on June 6th, 2014, and the subsequent announcement of the project in a June 10th [keynote by Google engineer Eric Brewer at DockerCon 2014](https://youtu.be/YrxnVKZeqK8?si=Q_wYBFn7dsS9H3k3) (and its corresponding [Google blog](https://cloudplatform.googleblog.com/2014/06/an-update-on-container-support-on-google-cloud-platform.html)). Over the next year, a small community of [contributors, largely from Google and Red Hat](https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=Before%20joining%20CNCF&var-metric=contributors), worked hard on the project, culminating in a [version 1.0 release on July 21st, 2015](https://cloudplatform.googleblog.com/2015/07/Kubernetes-V1-Released.html). Alongside 1.0, Google announced that Kubernetes would be donated to a newly formed branch of the Linux Foundation called the [Cloud Native Computing Foundation (CNCF)](https://www.cncf.io/announcements/2015/06/21/new-cloud-native-computing-foundation-to-drive-alignment-among-container-technologies/). Despite reaching 1.0, the Kubernetes project was still very challenging to use and understand. Kubernetes contributor Kelsey Hightower took special note of the project's shortcomings in ease of use and on July 7, 2016, he pushed the [first commit of his famed "Kubernetes the Hard Way" guide](https://github.com/kelseyhightower/kubernetes-the-hard-way/commit/9d7ace8b186f6ebd2e93e08265f3530ec2fba81c). The project has changed enormously since its original 1.0 release; experiencing a number of big wins such as [Custom Resource Definitions (CRD) going GA in 1.16](https://kubernetes.io/blog/2019/09/18/kubernetes-1-16-release-announcement/) or [full dual stack support launching in 1.23](https://kubernetes.io/blog/2021/12/08/dual-stack-networking-ga/) and community "lessons learned" from the [removal of widely used beta APIs in 1.22](https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/) or the deprecation of [Dockershim](https://kubernetes.io/blog/2020/12/02/dockershim-faq/). Some notable updates, milestones and events since 1.0 include: * December 2016 - [Kubernetes 1.5](https://kubernetes.io/blog/2016/12/kubernetes-1-5-supporting-production-workloads/) introduces runtime pluggability with initial CRI support and alpha Windows node support. OpenAPI also appears for the first time, paving the way for clients to be able to discover extension APIs. * This release also introduced StatefulSets and PodDisruptionBudgets in Beta. * April 2017 — [Introduction of Role-Based Access Controls or RBAC](https://kubernetes.io/blog/2017/04/rbac-support-in-kubernetes/). * June 2017 — In [Kubernetes 1.7](https://kubernetes.io/blog/2017/06/kubernetes-1-7-security-hardening-stateful-application-extensibility-updates/), ThirdPartyResources or "TPRs" are replaced with CustomResourceDefinitions (CRDs). * December 2017 — [Kubernetes 1.9](https://kubernetes.io/blog/2017/12/kubernetes-19-workloads-expanded-ecosystem/) sees the Workloads API becoming GA (Generally Available). The release blog states: *"Deployment and ReplicaSet, two of the most commonly used objects in Kubernetes, are now stabilized after more than a year of real-world use and feedback."* * December 2018 — In 1.13, the Container Storage Interface (CSI) reaches GA, kubeadm tool for bootstrapping minimum viable clusters reaches GA, and CoreDNS becomes the default DNS server. * September 2019 — [Custom Resource Definitions go GA](https://kubernetes.io/blog/2019/09/18/kubernetes-1-16-release-announcement/) in Kubernetes 1.16. * August 2020 — [Kubernetes 1.19](https://kubernetes.io/blog/2020/08/31/kubernetes-1-19-feature-one-year-support/) increases the support window for releases to 1 year. * December 2020 — [Dockershim is deprecated](https://kubernetes.io/blog/2020/12/18/kubernetes-1.20-pod-impersonation-short-lived-volumes-in-csi/) in 1.20 * April 2021 — the [Kubernetes release cadence changes](https://kubernetes.io/blog/2021/07/20/new-kubernetes-release-cadence/#:~:text=On%20April%2023%2C%202021%2C%20the,Kubernetes%20community's%20contributors%20and%20maintainers.) from 4 releases per year to 3 releases per year. * July 2021 — Widely used beta APIs are [removed](https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/) in Kubernetes 1.22. * May 2022 — Kubernetes 1.24 sees [beta APIs become disabled by default](https://kubernetes.io/blog/2022/05/03/kubernetes-1-24-release-announcement/) to reduce upgrade conflicts and removal of [Dockershim](https://kubernetes.io/dockershim), leading to [widespread user confusion](https://www.youtube.com/watch?v=a03Hh1kd6KE) (we've since [improved our communication!](https://github.com/kubernetes/community/tree/master/communication/contributor-comms)) * December 2022 — In 1.26, there was a significant batch and [Job API overhaul](https://kubernetes.io/blog/2022/12/29/scalable-job-tracking-ga/) that paved the way for better support for AI /ML / batch workloads. **PS:** Curious to see how far the project has come for yourself? Check out this [tutorial for spinning up a Kubernetes 1.0 cluster](https://github.com/spurin/kubernetes-v1.0-lab) created by community members Carlos Santana, Amim Moises Salum Knabben, and James Spurin. --- Kubernetes offers more extension points than we can count. Originally designed to work with Docker and only Docker, now you can plug in any container runtime that adheres to the CRI standard. There are other similar interfaces: CSI for storage and CNI for networking. And that's far from all you can do. In the last decade, whole new patterns have emerged, such as using [Custom Resource Definitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CRDs) to support third-party controllers - now a huge part of the Kubernetes ecosystem. The community building the project has also expanded immensely over the last decade. Using [DevStats](https://k8s.devstats.cncf.io/d/24/overall-project-statistics?orgId=1), we can see the incredible volume of contribution over the last decade that has made Kubernetes the [second-largest open source project in the world](https://www.cncf.io/reports/kubernetes-project-journey-report/): * **88,474** contributors * **15,121** code committers * **4,228,347** contributions * **158,530** issues * **311,787** pull requests ## **Kubernetes today** ![KubeCon NA 2023](https://kubernetes.io/blog/2024/06/06/10-years-of-kubernetes/welcome.jpg align="left") Since its early days, the project has seen enormous growth in technical capability, usage, and contribution. The project is still actively working to improve and better serve its users. In the upcoming 1.31 release, the project will celebrate the culmination of an important long-term project: the removal of in-tree cloud provider code. In this [largest migration in Kubernetes history](https://kubernetes.io/blog/2024/05/20/completing-cloud-provider-migration/), roughly 1.5 million lines of code have been removed, reducing the binary sizes of core components by approximately 40%. In the project's early days, it was clear that extensibility would be key to success. However, it wasn't always clear how that extensibility should be achieved. This migration removes a variety of vendor-specific capabilities from the core Kubernetes code base. Vendor-specific capabilities can now be better served by other pluggable extensibility features or patterns, such as [Custom Resource Definitions (CRDs)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) or API standards like the [Gateway API](https://gateway-api.sigs.k8s.io/). Kubernetes also faces new challenges in serving its vast user base, and the community is adapting accordingly. One example of this is the migration of image hosting to the new, community-owned registry.k8s.io. The egress bandwidth and costs of providing pre-compiled binary images for user consumption have become immense. This new registry change enables the community to continue providing these convenient images in more cost- and performance-efficient ways. Make sure you check out the [blog post](https://kubernetes.io/blog/2022/11/28/registry-k8s-io-faster-cheaper-ga/) and update any automation you have to use registry.k8s.io! ## **The future of Kubernetes** ![](https://kubernetes.io/blog/2024/06/06/10-years-of-kubernetes/lts.jpg align="left") A decade in, the future of Kubernetes still looks bright. The community is prioritizing changes that both improve the user experiences, and enhance the sustainability of the project. The world of application development continues to evolve, and Kubernetes is poised to change along with it. In 2024, the advent of AI changed a once-niche workload type into one of prominent importance. Distributed computing and workload scheduling has always gone hand-in-hand with the resource-intensive needs of Artificial Intelligence, Machine Learning, and High Performance Computing workloads. Contributors are paying close attention to the needs of newly developed workloads and how Kubernetes can best serve them. The new [Serving Working Group](https://github.com/kubernetes/community/tree/master/wg-serving) is one example of how the community is organizing to address these workloads' needs. It's likely that the next few years will see improvements to Kubernetes' ability to manage various types of hardware, and its ability to manage the scheduling of large batch-style workloads which are run across hardware in chunks. The ecosystem around Kubernetes will continue to grow and evolve. In the future, initiatives to maintain the sustainability of the project, like the migration of in-tree vendor code and the registry change, will be ever more important. The next 10 years of Kubernetes will be guided by its users and the ecosystem, but most of all, by the people who contribute to it. The community remains open to new contributors. You can find more information about contributing in our New Contributor Course at [https://k8s.dev/docs/onboarding](https://k8s.dev/docs/onboarding). We look forward to building the future of Kubernetes with you! ![KCSNA 2023](https://kubernetes.io/blog/2024/06/06/10-years-of-kubernetes/kcsna2023.jpg align="left")