# Storage

> Cordium documentation. Canonical page: <https://octelium.com/docs/cordium/latest/management/storage>.

Cordium stores everything on Kubernetes persistent volumes provisioned by your cluster's CSI drivers. This page explains how each Cordium feature maps to Kubernetes storage, how to choose the storage backends, and how to troubleshoot storage issues. For the installation requirements, read [here](https://octelium.com/docs/cordium/latest/install/cluster.md#storage).

## How Storage Is Used

All the storage objects live in the `cordium` Kubernetes namespace:

| Cordium feature                                | Kubernetes objects                                                                                                                                                                                                                                                      | Requirement                                                                                                       |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| *Workspace* storage                            | A `ReadWriteOnce` `PersistentVolumeClaim` named `ws-<WORKSPACE_UID>` per *Workspace*, sized by the *Workspace*'s storage limit. It is kept while a persistent *Workspace* is stopped, and deleted when an ephemeral *Workspace* stops or when a *Workspace* is deleted. | A `StorageClass` with dynamic provisioning. **Required.**                                                         |
| `ACCESS_MODE_EXCLUSIVE` *Volumes*              | A `ReadWriteOnce` `PersistentVolumeClaim` named `vol-<VOLUME_UID>` per *Volume*.                                                                                                                                                                                        | The same as above.                                                                                                |
| `ACCESS_MODE_SHARED` *Volumes*                 | A `ReadWriteMany` `PersistentVolumeClaim` named `vol-<VOLUME_UID>` per *Volume*.                                                                                                                                                                                        | A `StorageClass` of a multi-writer filesystem backend. Optional.                                                  |
| *Template* pre-builds and *WorkspaceSnapshots* | A `VolumeSnapshot` of the *Workspace*'s claim, from which new claims are restored.                                                                                                                                                                                      | The `VolumeSnapshot` CRDs, the CSI snapshot controller, and a `VolumeSnapshotClass` for the CSI driver. Optional. |
| Growing *Volumes*                              | An expansion of the *Volume*'s claim.                                                                                                                                                                                                                                   | A `StorageClass` with `allowVolumeExpansion: true`. Optional.                                                     |

Since each *Workspace* claim is attached to a single pod, block storage is the best fit for *Workspaces*: it is fast, and most block storage CSI drivers support copy-on-write snapshots, which makes pre-builds and *WorkspaceSnapshots* cheap and fast to take and to restore.

## Choosing Storage Backends

Here are some common setups:

| Environment                     | *Workspaces* and `EXCLUSIVE` *Volumes*                                                                            | `SHARED` *Volumes*                                       |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| Single-node or on-prem clusters | [Longhorn](https://longhorn.io) (installed by the quick installer's `--longhorn` flag), OpenEBS or Rook-Ceph RBD. | Longhorn RWX volumes, Rook-CephFS or the NFS CSI driver. |
| AWS EKS                         | The EBS CSI driver (e.g. `gp3`).                                                                                  | The EFS CSI driver.                                      |
| Google GKE                      | The Persistent Disk CSI driver (e.g. `pd-balanced` or `pd-ssd`).                                                  | Filestore CSI driver.                                    |
| Azure AKS                       | The Azure Disk CSI driver.                                                                                        | The Azure Files CSI driver.                              |

You can verify that the snapshot support is installed in your Kubernetes cluster as follows:

```bash
kubectl get crd volumesnapshots.snapshot.storage.k8s.io
kubectl get volumesnapshotclass
```

## Selecting Storage Classes

If your Kubernetes cluster has several `StorageClasses`, you can choose the class of each *Workspace* and *Volume*, as well as the `VolumeSnapshotClass` of the snapshots, via CEL rules in the `ClusterConfig`. Here is an example for an EKS cluster that uses fast `io2` volumes for large *Workspaces*, `gp3` volumes for everything else and EFS for the `SHARED` *Volumes*:

```yaml
kind: ClusterConfig
spec:
  workspace:
    storage:
      storageClass:
        rules:
          - condition:
              match: ctx.workspace.status.limit.storage.megabytes >= 100000
            storageClass: ebs-io2
          - condition:
              matchAny: true
            storageClass: ebs-gp3
      volumeSnapshotClass:
        rules:
          - condition:
              matchAny: true
            volumeSnapshotClass: ebs-csi-snapclass
  volume:
    storage:
      storageClass:
        rules:
          - condition:
              match: ctx.volume.spec.accessMode == "ACCESS_MODE_SHARED"
            storageClass: efs-sc
          - condition:
              matchAny: true
            storageClass: ebs-gp3
```

You can read more about these rules [here](https://octelium.com/docs/cordium/latest/management/clusterconfig.md#storage).

> **Note:**
>
> A restored claim must use the same CSI driver as the snapshot that it is restored from. If you route *Workspaces* to `StorageClasses` of different CSI drivers, make sure that the *Workspaces* restored from a *Template* pre-build or a *WorkspaceSnapshot* are routed to the same driver as their source.

## Regions

Every claim and snapshot belongs to the *Region* in which it was created. Cordium therefore runs a persistent *Workspace* in the *Region* of its storage, a *Workspace* that mounts *Volumes* in the *Region* of its *Volumes*, and a restored *Workspace* in the *Region* of its snapshot. The *Region* of a *Volume* can be chosen at creation time (e.g. `cordium create volume --region eu-west`), and it otherwise defaults to the creator's preferred *Region*.

## Troubleshooting

Here are the most common storage issues:

- **A *Workspace* is stuck while initializing.** The claim is typically waiting for the storage to be provisioned. Inspect it via `kubectl -n cordium describe pvc ws-<WORKSPACE_UID>`. You can get the *Workspace*'s UID via `cordium get ws <NAME> -o yaml`.
- **A *Volume* stays `STATE_PENDING`.** This is expected until the first *Workspace* that mounts it is scheduled with storage backends that delay the binding until the first consumer. If a *Workspace* that mounts it fails to start, inspect the claim via `kubectl -n cordium describe pvc vol-<VOLUME_UID>`. This is typically caused by a `SHARED` *Volume* whose `StorageClass` cannot provision `ReadWriteMany` volumes.
- **A *WorkspaceSnapshot* fails with an `Unsupported` failure.** The `VolumeSnapshot` API is not installed or no `VolumeSnapshotClass` is available for the CSI driver. Install the CSI snapshot controller and a `VolumeSnapshotClass`, or select one explicitly via the `ClusterConfig`.
- **A *Volume* cannot be grown.** The `StorageClass` must have `allowVolumeExpansion: true`, and some drivers only expand volumes that are not attached to a running pod.