# Local Storage and Kubernetes Guides Using local storage with OpenEBS Jiva If you want to use replicated storage leveraging disk space from a local disk with Talos Linux installed, OpenEBS Jiva is a great option. This requires installing the [iscsi-tools](https://github.com/orgs/siderolabs/packages?tab=packages&q=iscsi-tools) [system extension](https://www.talos.dev/v1.1/talos-guides/configuration/system-extensions/). Since OpenEBS Jiva is a replicated storage, it’s recommended to have at least three nodes where sufficient local disk space is available. The documentation will follow installing OpenEBS Jiva via the offical Helm chart. Since Talos is different from standard Operating Systems, the OpenEBS components need a little tweaking after the Helm installation. Refer to the OpenEBS Jiva [documentation](https://github.com/openebs/jiva-operator/blob/develop/docs/quickstart.md) if you need further customization. > NB: Also note that the Talos nodes need to be upgraded with `--preserve` set while running OpenEBS Jiva, otherwise you risk losing data. Even though it’s possible to recover data from other replicas if the node is wiped during an upgrade, this can require extra operational knowledge to recover, so it’s highly recommended to use `--preserve` to avoid data loss. ## **Preparing the nodes** Create a machine config patch with the contents below and save as `patch.yaml` ```yaml - op: add path: /machine/install/extensions value: - image: ghcr.io/siderolabs/iscsi-tools:v0.1.1 - op: add path: /machine/kubelet/extraMounts value: - destination: /var/openebs/local type: bind source: /var/openebs/local options: - bind - rshared - rw ``` Apply the machine config to all the nodes using talosctl: ```bash talosctl -e -n patch mc -p @patch.yaml ``` To install the system extension, the node needs to be upgraded. If there is no new release of Talos, the node can be upgraded to the same version as the existing Talos version. Run the following command on each nodes subsequently: ```bash talosctl -e -n upgrade --image=ghcr.io/siderolabs/installer:v1.1.1 ``` Once the node has upgraded and booted successfully the extension status can be verfied by running the following command: ```bash talosctl -e -n get extensions ``` An output similar to below can be observed: ```text NODE NAMESPACE TYPE ID VERSION NAME VERSION 192.168.20.61 runtime ExtensionStatus 000.ghcr.io-siderolabs-iscsi-tools-v0.1.1 1 iscsi-tools v0.1.1 ``` The service status can be checked by running the following command: ```bash talosctl -e -n services ``` You should see that the `ext-tgtd` and the `ext-iscsid` services are running. ```text NODE SERVICE STATE HEALTH LAST CHANGE LAST EVENT 192.168.20.51 apid Running OK 64h57m15s ago Health check successful 192.168.20.51 containerd Running OK 64h57m23s ago Health check successful 192.168.20.51 cri Running OK 64h57m20s ago Health check successful 192.168.20.51 etcd Running OK 64h55m29s ago Health check successful 192.168.20.51 ext-iscsid Running ? 64h57m19s ago Started task ext-iscsid (PID 4040) for container ext-iscsid 192.168.20.51 ext-tgtd Running ? 64h57m19s ago Started task ext-tgtd (PID 3999) for container ext-tgtd 192.168.20.51 kubelet Running OK 38h14m10s ago Health check successful 192.168.20.51 machined Running ? 64h57m29s ago Service started as goroutine 192.168.20.51 trustd Running OK 64h57m19s ago Health check successful 192.168.20.51 udevd Running OK 64h57m21s ago Health check successful ``` ## **Install OpenEBS Jiva** ```bash helm repo add openebs-jiva https://openebs.github.io/jiva-operator helm repo update helm upgrade --install --create-namespace --namespace openebs --version 3.2.0 openebs-jiva openebs-jiva/jiva ``` This will create a storage class named `openebs-jiva-csi-default` which can be used for workloads. The storage class named `openebs-hostpath` is used by jiva to create persistent volumes backed by local storage and then used for replicated storage by the jiva controller. ## **Patching the jiva installation** Since Jiva assumes `iscisd` to be running natively on the host and not as a Talos [extension service](https://www.talos.dev/v1.1/advanced/extension-services/), we need to modify the CSI node daemonset to enable it to find the PID of the `iscsid` service. The default config map used by Jiva also needs to be modified so that it can execute `iscsiadm` commands inside the PID namespace of the `iscsid` service. Start by creating a configmap definition named `config.yaml` as below: ```yaml apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/managed-by: pulumi name: openebs-jiva-csi-iscsiadm namespace: openebs data: iscsiadm: | #!/bin/sh iscsid_pid=$(pgrep iscsid) nsenter --mount="/proc/${iscsid_pid}/ns/mnt" --net="/proc/${iscsid_pid}/ns/net" -- /usr/local/sbin/iscsiadm "$@" ``` Replace the existing config map with the above config map by running the following command: ```bash kubectl --namespace openebs apply --filename config.yaml ``` Now we need to update the jiva CSI daemonset to run with `hostPID: true` so it can find the PID of the `iscsid` service, by running the following command: ```bash kubectl --namespace openebs patch daemonset openebs-jiva-csi-node --type=json --patch '[{"op": "add", "path": "/spec/template/spec/hostPID", "value": true}]' ``` ## **Testing a simple workload** In order to test the Jiva installation, let’s first create a PVC referencing the `openebs-jiva-csi-default` storage class: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: example-jiva-csi-pvc spec: storageClassName: openebs-jiva-csi-default accessModes: - ReadWriteOnce resources: requests: storage: 4Gi ``` and then create a deployment using the above PVC: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: fio spec: selector: matchLabels: name: fio replicas: 1 strategy: type: Recreate rollingUpdate: null template: metadata: labels: name: fio spec: containers: - name: perfrunner image: openebs/tests-fio command: ["/bin/bash"] args: ["-c", "while true ;do sleep 50; done"] volumeMounts: - mountPath: /datadir name: fio-vol volumes: - name: fio-vol persistentVolumeClaim: claimName: example-jiva-csi-pvc ``` You can clean up the test resources by running the following command: ```bash kubectl delete deployment fio kubectl delete pvc example-jiva-csi-pvc ```