# True Secrets Auto Rotation with ESO and Vault *Translation to pt-br at* [*https://knela.dev/blog/auto-rotao-de-segredos-de-verdade-com-eso-e-vault*](https://knela.dev/blog/auto-rotao-de-segredos-de-verdade-com-eso-e-vault) ## Requirements * A Kubernetes cluster that you can use (kind, minikube, something managed) and kubectl to connect to it * Vault CLI * External Secrets Operator ([ESO](https://external-secrets.io/)) [installed](https://artifacthub.io/packages/helm/external-secrets-operator/external-secrets). * Vault installed through the [helm chart](https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-raft) ## What we want to achieve [![Image description](https://res.cloudinary.com/practicaldev/image/fetch/s--T12pVZgu--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vk3769u373fiungnrn8e.png align="left")](https://res.cloudinary.com/practicaldev/image/fetch/s--T12pVZgu--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vk3769u373fiungnrn8e.png) This guide aims to establish an automatic hourly rotation of a database connection secret. Following these steps, an administrator sets up the process once, ensuring that the secret refreshes/changes every hour. Simultaneously, the application will always maintain new valid credentials for seamless database interactions. [![Image description](https://res.cloudinary.com/practicaldev/image/fetch/s--cpf8f1aO--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ughfgbdpt2nrtjjlvpzu.png align="left")](https://res.cloudinary.com/practicaldev/image/fetch/s--cpf8f1aO--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ughfgbdpt2nrtjjlvpzu.png) ## ESO secret [Generators](https://external-secrets.io/v0.8.5/api/generator/vault/) ⚠️ As of this writing this feature is in alpha state, and we want more people to help test it, so we can make improvements, and eventually promote to stable. Documentation around it is a bit limited, that's why I am getting this guide out in my blog, while we figure better ways to bring these into our documentation. [![Image description](https://res.cloudinary.com/practicaldev/image/fetch/s--2LQhCsMw--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8u4nmhydcsg4mg3ly63c.png align="left")](https://res.cloudinary.com/practicaldev/image/fetch/s--2LQhCsMw--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8u4nmhydcsg4mg3ly63c.png) ## Getting Started Let's make sure we start with the same setup locally: * I have installed Vault in a namespace named vault. * You can use `helm install vault hashicorp/vault -n vault --create-namespace` command instead of the one provided [in the guide](https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-raft). * Follow all steps in there to init Vault, unseal, and get the `cluster-keys.json` with the token. * You can skip other steps. * I have [installed ESO](https://artifacthub.io/packages/helm/external-secrets-operator/external-secrets) in the default namespace. In this guide we are going to use Vault token authentication just for the sake of simplicity. However, please never use this in real setups. Prefer service account auth. After properly starting Vault and unsealing it, take note of your auth token. Let's do a port forward and authenticate in our work desktop so we don't have to exec into Vault every time we need to run commands. In a new terminal (this terminal will be blocked) ```less kubectl -n vault port-forward service/vault 8200:8200 ``` In another terminal you can run. ```less export VAULT_ADDR=http://127.0.0.1:8200 vault login ## type your auth token ``` [![Image description](https://res.cloudinary.com/practicaldev/image/fetch/s--a4fEXqGk--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ayv1stf8hjfb135mmil1.png align="left")](https://res.cloudinary.com/practicaldev/image/fetch/s--a4fEXqGk--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ayv1stf8hjfb135mmil1.png) ## Simple Deployment of PostgreSQL To have an interesting example, let's deploy psql and configure it so we can let Vault and other workloads connect to it. Let's first create a configmap with an admin user and password for this psql instance (just for simplicity and to get to the other part of the guide quickly). ```less cat < postgres-config.yaml apiVersion: v1 kind: ConfigMap metadata: name: postgres-config labels: app: postgres data: POSTGRES_DB: postgresdb POSTGRES_USER: admin POSTGRES_PASSWORD: psltest EOF ``` Apply it. ```less kubectl apply -f postgres-config.yaml ``` Now create the postgres-deployment.yaml. ```less cat < postgres-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: postgres # Sets Deployment name spec: replicas: 1 selector: matchLabels: app: postgres template: metadata: labels: app: postgres spec: containers: - name: postgres image: postgres:10.1 # Sets Image imagePullPolicy: "IfNotPresent" ports: - containerPort: 5432 # Exposes container port envFrom: - configMapRef: name: postgres-config EOF ``` Apply it. ```less kubectl apply -f postgres-deployment.yaml ``` And finally, let's create a service, so other workloads can access it: ```less cat < postgres-service.yaml apiVersion: v1 kind: Service metadata: name: postgres # Sets service name labels: app: postgres # Labels and Selectors spec: type: NodePort # Sets service type ports: - port: 5432 # Sets port to run the postgres application selector: app: postgres EOF ``` Apply it. ```less kubectl apply -f postgres-service.yaml ``` ## Preparing DB with new readonly role Exec into the psql pod. ```less kubectl get pods # get pod name kubectl exec -it -- bash ``` Change into postgres user, and run commands to create the new role. ```less su postgres psql -c "CREATE ROLE \"ro\" NOINHERIT;" psql -c "GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"ro\";" ``` We are going to use this role when configuring Vault to use Dynamic Secrets with psql plugin. [![Image description](https://res.cloudinary.com/practicaldev/image/fetch/s--fOuMw2zs--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r74yrbcr68cocp3v0vlj.jpg align="left")](https://res.cloudinary.com/practicaldev/image/fetch/s--fOuMw2zs--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r74yrbcr68cocp3v0vlj.jpg) ## Vault Dynamic Secrets Vault Dynamic secrets are in fact meant to be used as a way to get short lived credentials. However, there is nothing stopping us from using them in our auto-rotation process. There are various other plugins that integrate with other systems, like AWS credentials, or certificate issuing systems. Most of these are also interesting in the context of ESO, but I wanted a self-contained example with no need to create external accounts for you to try it out. Lets first enable the database engine. ```less vault secrets enable database ``` After that, let's configure PostgreSQL secrets engine, with the admin creds we had before (we are passing credentials into the connection url here, never do that outside of test labs). ```less ## POSTGRES_URL with name of the service and namespace export POSTGRES_URL=postgres.default.svc.cluster.local:5432 vault write database/config/postgresql \ plugin_name=postgresql-database-plugin \ connection_url="postgresql://admin:psltest@$POSTGRES_URL/postgres?sslmode=disable" \ allowed_roles=readonly \ username="root" \ password="rootpassword" ``` Create an SQL file container the templated command that will be used by Vault when dynamically creating roles. ```less tee readonly.sql < vaultDynamicSecret.yaml apiVersion: generators.external-secrets.io/v1alpha1 kind: VaultDynamicSecret metadata: name: "psql-example" spec: path: "/database/creds/readonly" ## this is how you choose which vault dynamic path to use method: "GET" ## this path will only work with GETs # parameters: ## no needed parameters # ... provider: server: "http://vault.vault.svc.cluster.local:8200" ## vault url. In this case vault service on the vault namespace auth: # points to a secret that contains a vault token # https://www.vaultproject.io/docs/auth/token tokenSecretRef: ## reference to the secret holding the Vault auth token name: "vault-token" key: "token" --- apiVersion: v1 kind: Secret metadata: name: vault-token data: token: aHZzLkM4M0o2UWNQSW1YQkRJVU96aWNNNzVHdwo= ## token base64 encoded EOF ``` Apply this file. ```less kubectl apply -f vaultDynamicSecret.yaml ``` And finally we can now create our ExternalSecret that in the end will let the operator create the final Kubernetes Secret. ```less cat < vaultDynamicSecret.yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: "psql-example-es" spec: refreshInterval: "1h" ## the same as the expiry time on the dynamic config of Vault, or lower, so apps have always new valid credentials target: name: psql-example-for-use ## the final name of the kubernetes secret created in your cluster dataFrom: - sourceRef: generatorRef: apiVersion: generators.external-secrets.io/v1alpha1 kind: VaultDynamicSecret name: "psql-example" ## reference to the generator EOF ``` Apply this and check if the status of the ExternalSecret is ok. ```less kubectl get externalsecret ## response NAME STORE REFRESH INTERVAL STATUS READY psql-example-es 1h SecretSynced True ``` If you get errors here, verify that you used the right path in the Generator. Also check that you created the right roles inside psql and you can ping vault from a pod in the ESO namespace. ## Checking the final secret You should get a secret containing new users and passwords with read-only access to the database every hour. ```less kubectl get secrets psql-example-for-use -o jsonpath="{.data}" ## response {"password":"V2lSWUlqZzdvQS1yOTFaV2N1SWE=","username":"di1yb290LXJlYWRvbmx5LVlXQ3kzZ01hbkhSbGtuY3FqTUg2LTE2OTA0NzIwMzc="} ``` To check one of the values you can get it and base64 decode it. ```less kubectl get secrets psql-example-for-use -o jsonpath="{.data.password}" | base64 -d ``` Now your application can use this secret, it will be automatically auto-rotated, and still be a valid credential to the database. ## Caveats If you use secrets as Environment Variables you will need to use something to make workloads get the new credentials, if they just loose connection. You can use the [Reloader](https://github.com/stakater/Reloader) project for that. If you use secrets as volumes, pods will get that update automatically, and you won't have problems connecting, as long as your application can get the new values. ## Conclusion [![Image description](https://res.cloudinary.com/practicaldev/image/fetch/s--Pb07phGh--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x079b7hakbqw2qhex5qk.jpg align="left")](https://res.cloudinary.com/practicaldev/image/fetch/s--Pb07phGh--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x079b7hakbqw2qhex5qk.jpg) That's it! We've set up an auto-rotating secret for a database connection using ESO and Vault. The magic is, as we said, you can set it once and forget. Your secret refreshes every hour and your app stays connected to the database with new valid credentials. It is secure, you follow best practices with regard to rotation, and you avoid manual intervention if that is not needed.