# Prometheus-based Kubernetes Resource Recommendations


![Product Name Screen Shot](https://github.com/robusta-dev/krr/raw/main/images/screenshot.jpeg align="center")

## Get recommendations based on your existing data in Prometheus/Coralogix/Thanos/Mimir and more!

## About The Project

Robusta KRR (Kubernetes Resource Recommender) is a CLI tool for **optimizing resource allocation** in Kubernetes clusters. It gathers pod usage data from Prometheus and **recommends requests and limits** for CPU and memory. This **reduces costs and improves performance**.

### Data Integrations

[![Used to send data to KRR](https://github.com/robusta-dev/krr/raw/main/images/krr-datasources.svg align="left")](https://github.com/robusta-dev/krr#data-source-integrations)

*View Instructions for:* [*Prometheus*](https://github.com/robusta-dev/krr#prometheus-victoria-metrics-and-thanos-auto-discovery)*,* [*Thanos*](https://github.com/robusta-dev/krr#prometheus-victoria-metrics-and-thanos-auto-discovery)*,* [*Victoria Metrics*](https://github.com/robusta-dev/krr#prometheus-victoria-metrics-and-thanos-auto-discovery)*,* [*Google Managed Prometheus*](https://github.com/robusta-dev/krr/blob/main/docs/google-cloud-managed-service-for-prometheus.md)*,* [*Amazon Managed Prometheus*](https://github.com/robusta-dev/krr#amazon-managed-prometheus)*,* [*Azure Managed Prometheus*](https://github.com/robusta-dev/krr#azure-managed-prometheus)*,* [*Coralogix*](https://github.com/robusta-dev/krr#coralogix-managed-prometheus)*,*[*Grafana Cloud*](https://github.com/robusta-dev/krr#grafana-cloud-managed-prometheus) *and* [*Grafana Mimir*](https://github.com/robusta-dev/krr#grafana-mimir-auto-discovery)

### Reporting Integrations

[![Used to receive information from KRR](https://github.com/robusta-dev/krr/raw/main/images/krr-other-integrations.svg align="left")](https://github.com/robusta-dev/krr#integrations)

*View instructions for:* [*Seeing recommendations in a UI*](https://github.com/robusta-dev/krr#free-ui-for-krr-recommendations)*,* [*Sending recommendations to Slack*](https://github.com/robusta-dev/krr#slack-notification)*,* [*Setting up KRR as a k9s plugin*](https://github.com/robusta-dev/krr#k9s-plugin)

### Features

* **No Agent Required**: Run a CLI tool on your local machine for immediate results. (Or run in-cluster for weekly [Slack reports](https://github.com/robusta-dev/krr#slack-integration).)
    
* **Prometheus Integration**: Get recommendations based on the data you already have
    
* **Explainability**: [Understand how recommendations were calculated with explanation graphs](https://github.com/robusta-dev/krr#free-krr-ui-on-robusta-saas)
    
* **Extensible Strategies**: Easily create and use your own strategies for calculating resource recommendations.
    
* **Free SaaS Platform**: See why KRR recommends what it does, by using the [free Robusta SaaS platform](https://platform.robusta.dev/signup/?utm_source=github&utm_medium=krr-readme).
    
* **Future Support**: Upcoming versions will support custom resources (e.g. GPUs) and custom metrics.
    

### How Much Can I Expect to Save with KRR?

According to a recent [Sysdig study](https://sysdig.com/blog/millions-wasted-kubernetes/), on average, Kubernetes clusters have:

* 69% unused CPU
    
* 18% unused memory
    

By right-sizing your containers with KRR, you can save an average of 69% on cloud costs.

Read more about [how KRR works](https://github.com/robusta-dev/krr#how-krr-works)

## Difference with Kubernetes VPA

| Feature 🛠️ | Robusta KRR 🚀 | Kubernetes VPA 🌐 |
| --- | --- | --- |
| Resource Recommendations 💡 | ✅ CPU/Memory requests and limits | ✅ CPU/Memory requests and limits |
| Installation Location 🌍 | ✅ Not required to be installed inside the cluster, can be used on your own device, connected to a cluster | ❌ Must be installed inside the cluster |
| Workload Configuration 🔧 | ✅ No need to configure a VPA object for each workload | ❌ Requires VPA object configuration for each workload |
| Immediate Results ⚡ | ✅ Gets results immediately (given Prometheus is running) | ❌ Requires time to gather data and provide recommendations |
| Reporting 📊 | ✅ Json, CSV, Markdown, [Web UI](https://github.com/robusta-dev/krr#free-ui-for-krr-recommendations), and more! | ❌ Not supported |
| Extensibility 🔧 | ✅ Add your own strategies with few lines of Python | ⚠️ Limited extensibility |
| Explainability 📖 | ✅ [See graphs explaining the recommendations](https://github.com/robusta-dev/krr#free-krr-ui-on-robusta-saas) | ❌ Not supported |
| Custom Metrics 📏 | 🔄 Support in future versions | ❌ Not supported |
| Custom Resources 🎛️ | 🔄 Support in future versions (e.g., GPU) | ❌ Not supported |
| Autoscaling 🔀 | 🔄 Support in future versions | ✅ Automatic application of recommendations |
| Default History 🕒 | 14 days | 8 days |
| Supports HPA 🔥 | ✅ Enable using `--allow-hpa` flag | ❌ Not supported |

## Installation

### Requirements

KRR requires Prometheus 2.26+, [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) & [cAdvisor](https://github.com/google/cadvisor).

<details><summary>Which metrics does KRR need?</summary><a href="https://docs.robusta.dev/master/configuration/alertmanager-integration/embedded-prometheus.html"></a><p></p><ul><li><code></code></li><li><code></code></li><li><code></code></li><li><code></code></li><li><code></code></li></ul><p><em></em></p></details>

### Installation Methods

<details><summary>Brew (Mac/Linux)</summary><ol><li></li></ol><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><ol><li></li></ol><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><ol><li></li></ol><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><ol><li></li></ol><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div></details>

<details><summary>Windows</summary><p><a href="https://docs.brew.sh/Homebrew-on-Linux"></a></p></details>

<details><summary>Airgapped Installation (Offline Environments)</summary><p><a href="https://github.com/robusta-dev/krr/releases"></a></p><div class="snippet-clipboard-content notranslate position-relative overflow-auto"><pre><code></code></pre><div class="zeroclipboard-container"></div></div><p><strong></strong></p></details>

<details><summary>From Source</summary><ol><li><a href="https://www.python.org/downloads/"></a></li><li></li></ol><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><ol><li><code></code></li><li></li></ol><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><ol><li></li></ol><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><p><code></code><code></code><code></code></p></details>

### Additional Options

* [View KRR Reports in a Web UI](https://github.com/robusta-dev/krr#free-ui-for-krr-recommendations)
    
* [Receive KRR Reports Weekly in Slack](https://github.com/robusta-dev/krr#slack-notification)
    

### Environment-Specific Instructions

Setup KRR for...

* [Google Cloud Managed Prometheus](https://github.com/robusta-dev/krr/blob/main/docs/google-cloud-managed-service-for-prometheus.md)
    
* [Azure Managed Prometheus](https://github.com/robusta-dev/krr#azure-managed-prometheus)
    
* [Amazon Managed Prometheus](https://github.com/robusta-dev/krr#amazon-managed-prometheus)
    
* [Coralogix Managed Prometheus](https://github.com/robusta-dev/krr#coralogix-managed-prometheus)
    
* [Grafana Cloud Managed Prometheus](https://github.com/robusta-dev/krr#grafana-cloud-managed-prometheus)
    
* [Grafana Mimir](https://github.com/robusta-dev/krr#grafana-mimir-auto-discovery)
    

**Trusting custom Certificate Authority (CA) certificate:**

If your llm provider url uses a certificate from a custom CA, in order to trust it, base-64 encode the certificate, and store it in an environment variable named `CERTIFICATE`

## Free KRR UI on Robusta SaaS

We highly recommend using the [free Robusta SaaS platform](https://platform.robusta.dev/signup/?utm_source=github&utm_medium=krr-readme). You can:

* Understand individual app recommendations with app usage history
    
* Sort and filter recommendations by namespace, priority, and more
    
* Give devs a YAML snippet to fix the problems KRR finds
    
* Analyze impact using KRR scan history
    
    ![](https://camo.githubusercontent.com/eeb140ec20f95f9ab831f63b87009acc81b766f69b28b707d84926a8a9d8b698/68747470733a2f2f63646e2e6c6f6f6d2e636f6d2f73657373696f6e732f7468756d626e61696c732f34396561313838663637633534356630616539383530386234343864316138622d776974682d706c61792e676966 align="left")
    

## Usage

<details><summary>Basic usage</summary><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div></details>

<details><summary>Tweak the recommendation algorithm (strategy)</summary><p></p><ul><li><code></code></li><li><code></code></li><li><code></code></li></ul><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div></details>

<details><summary>Giving an Explicit Prometheus URL</summary><p><code></code></p><p><code></code></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div></details>

<details><summary>Run on specific namespaces</summary><p><code></code><code></code><code></code></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><p><a href="https://github.com/robusta-dev/krr/blob/main/tests/single_namespace_permissions.yaml"></a></p></details>

<details><summary>Run on workloads filtered by label</summary><p><a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#api"></a></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre><span class="pl-s"><span class="pl-pds"></span><span class="pl-pds"></span></span></pre><div class="zeroclipboard-container"></div></div></details>

<details><summary>Override the kubectl context</summary><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div></details>

<details><summary>Output formats for reporting (JSON, YAML, CSV, and more)</summary><p></p><ul><li><code></code><a href="https://github.com/Textualize/rich"></a></li><li><code></code></li><li><code></code></li><li><code></code></li><li><code></code></li></ul><p><code></code><code></code></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><p><code></code></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre><span class="pl-k"></span><span class="pl-k"></span></pre><div class="zeroclipboard-container"></div></div></details>

<details><summary>Centralized Prometheus (multi-cluster)</summary><p><a href="https://github.com/robusta-dev/krr#scanning-with-a-centralized-prometheus"></a></p></details>

<details><summary>Prometheus Authentication</summary><p></p><p><code></code><code></code><code></code><code></code><code></code><code></code><code></code></p><p></p></details>

<details><summary>Debug mode</summary><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div></details>

## How KRR works

### Metrics Gathering

Robusta KRR uses the following Prometheus queries to gather usage data:

* CPU Usage:
    
    ```basic
    sum(irate(container_cpu_usage_seconds_total{{namespace="{object.namespace}", pod="{pod}", container="{object.container}"}}[{step}]))
    ```
    
* Memory Usage:
    
    ```basic
    sum(container_memory_working_set_bytes{job="kubelet", metrics_path="/metrics/cadvisor", image!="", namespace="{object.namespace}", pod="{pod}", container="{object.container}"})
    ```
    

[*Need to customize the metrics? Tell us and we'll add support.*](https://github.com/robusta-dev/krr/issues/new)

Get a free breakdown of KRR recommendations in the [Robusta SaaS](https://github.com/robusta-dev/krr#free-krr-ui-on-robusta-saas).

### Algorithm

By default, we use a *simple* strategy to calculate resource recommendations. It is calculated as follows (*The exact numbers can be customized in CLI arguments*):

* For CPU, we set a request at the 95th percentile with no limit. Meaning, in 95% of the cases, your CPU request will be sufficient. For the remaining 5%, we set no limit. This means your pod can burst and use any CPU available on the node - e.g. CPU that other pods requested but aren’t using right now.
    
* For memory, we take the maximum value over the past week and add a 15% buffer.
    

### Prometheus connection

Find about how KRR tries to find the default Prometheus to connect [here](https://github.com/robusta-dev/krr#prometheus-victoria-metrics-and-thanos-auto-discovery).

## Data Source Integrations

<details id="user-content-prometheus-victoria-metrics-and-thanos-auto-discovery"><summary>Prometheus, Victoria Metrics and Thanos auto-discovery</summary><p></p><div class="highlight highlight-source-python notranslate position-relative overflow-auto"><pre><span class="pl-s"></span><span class="pl-s"></span><span class="pl-s"></span><span class="pl-s"></span><span class="pl-s"></span><span class="pl-s"></span></pre><div class="zeroclipboard-container"></div></div><p></p><div class="highlight highlight-source-python notranslate position-relative overflow-auto"><pre><span class="pl-s"></span><span class="pl-s"></span><span class="pl-s"></span><span class="pl-s"></span></pre><div class="zeroclipboard-container"></div></div><p></p><div class="highlight highlight-source-python notranslate position-relative overflow-auto"><pre><span class="pl-s"></span><span class="pl-s"></span><span class="pl-s"></span><span class="pl-s"></span></pre><div class="zeroclipboard-container"></div></div><p><code></code></p><p><a href="https://github.com/robusta-dev/krr#readme-top"></a></p></details>

<details id="user-content-scanning-with-a-centralized-prometheus"><summary>Scanning with a Centralized Prometheus</summary><p></p><p><code></code></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre></pre><div class="zeroclipboard-container"></div></div><p><code></code></p></details>

<details id="user-content-azure-managed-prometheus"><summary>Azure Managed Prometheus</summary><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre><span class="pl-c"><span class="pl-c"></span></span><span class="pl-c"><span class="pl-c"></span></span><span class="pl-s"><span class="pl-pds"></span><span class="pl-pds"></span></span><span class="pl-k"></span><span class="pl-c1"></span><span class="pl-smi"></span></pre><div class="zeroclipboard-container"></div></div><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre><span class="pl-s"><span class="pl-pds"></span><span class="pl-smi"></span><span class="pl-pds"></span></span></pre><div class="zeroclipboard-container"></div></div><p><a href="https://github.com/robusta-dev/krr#scanning-with-a-centralized-prometheus"></a></p><p><a href="https://github.com/robusta-dev/krr#readme-top"></a></p></details>

<details id="user-content-google-managed-prometheus"><summary>Google Managed Prometheus (GMP)</summary><p><a href="https://github.com/robusta-dev/krr/blob/main/docs/google-cloud-managed-service-for-prometheus.md"></a></p><p><a href="https://github.com/robusta-dev/krr#readme-top"></a></p></details>

<details id="user-content-amazon-managed-prometheus"><summary>Amazon Managed Prometheus</summary><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre><span class="pl-s"><span class="pl-pds"></span><span class="pl-pds"></span></span></pre><div class="zeroclipboard-container"></div></div><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre><span class="pl-c"><span class="pl-c"></span></span><span class="pl-c"><span class="pl-c"></span></span><span class="pl-c"><span class="pl-c"></span></span><span class="pl-c"><span class="pl-c"></span></span><span class="pl-c"><span class="pl-c"></span></span></pre><div class="zeroclipboard-container"></div></div><p><a href="https://github.com/robusta-dev/krr#scanning-with-a-centralized-prometheus"></a></p><p><a href="https://github.com/robusta-dev/krr#readme-top"></a></p></details>

<details id="user-content-coralogix-managed-prometheus"><summary>Coralogix Managed Prometheus</summary><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre><span class="pl-s"><span class="pl-pds"></span><span class="pl-pds"></span></span></pre><div class="zeroclipboard-container"></div></div><p><a href="https://github.com/robusta-dev/krr#scanning-with-a-centralized-prometheus"></a></p><p><a href="https://github.com/robusta-dev/krr#readme-top"></a></p></details>

<details id="user-content-grafana-cloud-managed-prometheus"><summary>Grafana Cloud Managed Prometheus</summary><p><code></code></p><p></p><div class="highlight highlight-source-shell notranslate position-relative overflow-auto"><pre><span class="pl-smi"></span><span class="pl-s"><span class="pl-pds"></span><span class="pl-smi"></span><span class="pl-smi"></span><span class="pl-pds"></span></span></pre><div class="zeroclipboard-container"></div></div><p><a href="https://github.com/robusta-dev/krr#scanning-with-a-centralized-prometheus"></a></p><p><a href="https://github.com/robusta-dev/krr#readme-top"></a></p></details>

<details id="user-content-grafana-mimir-auto-discovery"><summary>Grafana Mimir auto-discovery</summary><p></p><p></p><div class="highlight highlight-source-python notranslate position-relative overflow-auto"><pre><span class="pl-s"></span></pre><div class="zeroclipboard-container"></div></div><p><a href="https://github.com/robusta-dev/krr#readme-top"></a></p></details>

## Integrations

<details id="user-content-free-ui-for-krr-recommendations"><summary>Free UI for KRR recommendations</summary><p><a href="https://platform.robusta.dev/signup/?utm_source=github&utm_medium=krr-readme"></a></p><ul><li><p></p></li><li><p></p></li><li><p></p></li><li><p></p><a href="https://www.loom.com/share/49ea188f67c545f0ae98508b448d1a8b"><img src="https://camo.githubusercontent.com/eeb140ec20f95f9ab831f63b87009acc81b766f69b28b707d84926a8a9d8b698/68747470733a2f2f63646e2e6c6f6f6d2e636f6d2f73657373696f6e732f7468756d626e61696c732f34396561313838663637633534356630616539383530386234343864316138622d776974682d706c61792e676966" /></a></li></ul></details>

<details id="user-content-slack-notification"><summary>Slack Notification</summary><p></p><p><a target="_blank" href="https://github.com/robusta-dev/krr/blob/main/images/krr_slack_example.png"><img src="https://github.com/robusta-dev/krr/raw/main/images/krr_slack_example.png" alt="Slack Screen Shot" /></a></p><div class="markdown-heading"><h3></h3><a id="user-content-prerequisites" class="anchor" href="https://github.com/robusta-dev/krr#prerequisites"></a></div><ul><li></li></ul><div class="markdown-heading"><h3></h3><a id="user-content-setup" class="anchor" href="https://github.com/robusta-dev/krr#setup"></a></div><ol><li><a href="https://docs.robusta.dev/master/installation.html"></a></li><li><code></code></li></ol><div class="snippet-clipboard-content notranslate position-relative overflow-auto"><pre><code></code></pre><div class="zeroclipboard-container"></div></div><ol><li><code></code></li></ol><p><a href="https://github.com/robusta-dev/krr#readme-top"></a></p></details>

<details id="user-content-k9s-plugin"><summary>k9s Plugin</summary><p></p><p><a href="https://github.com/derailed/k9s/blob/master/plugins/resource-recommendations.yaml"></a></p><p><a href="https://k9scli.io/topics/plugins/"></a></p></details>

## Creating a Custom Strategy/Formatter

Look into the [examples](https://github.com/robusta-dev/krr/tree/main/examples) directory for examples on how to create a custom strategy/formatter.

## Testing

*We use pytest to run tests.*

1. Install the project manually (see above)
    
2. Navigate to the project root directory
    
3. Install poetry ([https://python-poetry.org/docs/#installing-with-the-official-installer](https://python-poetry.org/docs/#installing-with-the-official-installer))
    
4. Install dev dependencies:
    

```basic
poetry install --group dev
```

5. Install robusta\_krr as editable dependency:
    

```basic
pip install -e .
```

6. Run the tests:
    

```basic
poetry run pytest
```

## Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
    
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
    
3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
    
4. Push to the Branch (`git push origin feature/AmazingFeature`)
    
5. Open a Pull Request