Reference Architecture: Virtual Storage

Purpose: For platform engineers, provides datastore configuration, storage policies, and performance tuning guidance.

Overview

Virtual storage connects the physical disks (see Physical Storage) to Kubernetes VMs through datastores, storage policies, and CSI drivers. openCenter uses vSphere CSI or OpenStack Cinder CSI to provision persistent volumes dynamically. This document covers the hypervisor-level storage configuration that supports those CSI drivers.

vSphere Storage Options

Option	Description	Best For
VMFS Datastore (local)	Local disks formatted as VMFS 6	Single-host labs, small clusters
VMFS Datastore (SAN)	SAN LUNs presented as VMFS 6	Enterprise shared storage
NFS Datastore	NFS v3/v4.1 exports from NAS	File-based workloads, simplicity
vSAN	Distributed storage across local disks	Hyper-converged, no external SAN

VMFS Datastore Configuration

Setting	Value
VMFS Version	6
Block Size	1 MB (default, supports up to 64 TB VMDK)
Space Reclamation	Automatic (UNMAP)
Multipathing	Round Robin (for SAN LUNs)
Path Switching	IOPS = 1 (switch after every I/O for best distribution)

Datastore Layout

Create separate datastores for different workload tiers:

Datastore Name	Backing	Size	Purpose
ds-etcd-nvme	NVMe SSD (RAID 1)	500 GB	Control plane VMs (etcd disks)
ds-vms-ssd	SSD (RAID 10)	4 TB	Worker VMs, container images
ds-iso	Any	100 GB	ISO images, templates

Separating etcd onto NVMe-backed storage ensures that worker VM I/O does not affect etcd fsync latency.

vSphere Storage Policies

Storage policies define the service level for VM disks. The vSphere CSI driver uses these policies when provisioning PersistentVolumes.

Policy Name	Datastore Tag	Encryption	Replication	Use Case
oc-etcd	tier-0-nvme	Optional	N/A (RAID 1)	etcd, control plane OS
oc-performance	tier-1-ssd	Optional	N/A (RAID 10)	Worker OS, application PVs
oc-capacity	tier-2-hdd	Optional	N/A (RAID 5/6)	Logs, metrics, backups

Tag datastores in vCenter and reference tags in storage policies. The CSI driver matches PVC storageClassName to the corresponding policy.

vSAN Configuration (If Used)

Setting	Value
vSAN Version	7.0 U3+ or 8.0
Disk Group Type	All-flash
Cache Tier	1× NVMe SSD per disk group (≥ 400 GB)
Capacity Tier	2–4× SSD per disk group
Deduplication & Compression	Enabled (all-flash only)
Erasure Coding	RAID 5/6 for capacity tier (4+ hosts)
FTT (Failures to Tolerate)	1 (RAID 1 mirror, 3-host minimum)
Encryption	At-rest encryption (optional, requires KMS)
Stretched Cluster	Not recommended for Kubernetes (latency sensitive)

vSAN requires a minimum of 3 hosts. Each host contributes local disks to the distributed datastore. The vSAN HCL must be checked for disk and controller compatibility before purchasing hardware.

OpenStack Storage (Cinder)

Cinder Backend Options

Backend	Protocol	Performance	Use Case
LVM (local)	iSCSI	Moderate	Small deployments, testing
Ceph RBD	RADOS	High	Production, distributed storage
NetApp ONTAP	iSCSI / NFS	High	Enterprise NAS/SAN integration
Pure Storage	iSCSI / FC	Very High	High-performance workloads

Ceph RBD is the most common production backend for OpenStack + Kubernetes deployments. It provides distributed replication without dedicated SAN hardware.

Cinder Volume Types

Volume Type	Backend	Properties	Use Case
oc-etcd	Ceph SSD pool	volume_backend_name=ceph-ssd	etcd, control plane
oc-performance	Ceph SSD pool	volume_backend_name=ceph-ssd	Worker disks, app PVs
oc-capacity	Ceph HDD pool	volume_backend_name=ceph-hdd	Logs, metrics, backups

Ceph Pool Configuration (If Using Ceph)

Pool	Device Class	Replication	PG Count	Purpose
ceph-ssd	SSD	3× replicated	Auto (PG autoscaler)	etcd, VMs, app PVs
ceph-hdd	HDD	3× replicated	Auto	Logs, metrics, backups

Set min_size=2 to allow I/O to continue with one OSD down. Set size=3 for three-way replication.

CSI Driver Integration

vSphere CSI

The vSphere CSI driver provisions VMDKs as PersistentVolumes. It requires:

Requirement	Value
vCenter credentials	Stored as Kubernetes Secret in vmware-system-csi namespace
Storage Policy	Must exist in vCenter and be referenced in StorageClass
VM Hardware	PVSCSI controller, Hardware Version 15+
Permissions	CSI-specific vCenter role (see vSphere CSI docs)

Example StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: oc-performance
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "oc-performance"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

OpenStack Cinder CSI

The Cinder CSI driver provisions Cinder volumes as PersistentVolumes. It requires:

Requirement	Value
OpenStack credentials	Stored as Kubernetes Secret (cloud-config)
Availability Zone	Must match Nova AZ for volume attachment
Volume Type	Must exist in Cinder

Example StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: oc-performance
provisioner: cinder.csi.openstack.org
parameters:
  type: oc-performance
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

In-Cluster Storage (Longhorn)

openCenter deploys Longhorn as an in-cluster distributed storage engine. Longhorn runs on Kubernetes worker nodes and uses their local disks to provide replicated block storage.

Setting	Value
Replica Count	3
Data Locality	Best effort
Backup Target	S3-compatible (MinIO, Harbor) or NFS
Disk Path	/var/lib/longhorn on worker nodes

Longhorn is complementary to the hypervisor-level CSI driver. Use the CSI driver for VM-level disks and Longhorn for application-level PersistentVolumes that need replication across Kubernetes nodes.

Performance Tuning

Tuning	vSphere	OpenStack
Disk I/O scheduler	noop or none in guest OS	none in guest OS
Queue depth	PVSCSI default (64)	virtio-blk default (128)
Read-ahead	256 KB for sequential workloads	256 KB
Disk alignment	Automatic with VMFS 6	Automatic with virtio
Thin provisioning	Enabled (default)	Enabled (Cinder default)

Set the guest OS I/O scheduler to none when running on a hypervisor. The hypervisor and storage controller handle scheduling; a guest-level scheduler adds latency.

Considerations

• Thin vs. thick provisioning: Thin provisioning saves space but can cause out-of-space conditions if the datastore fills. Monitor datastore free space and alert at 20% remaining.
• Snapshots: VM-level snapshots (vSphere/Cinder) are not a backup strategy. They consume growing delta disks and degrade I/O performance. Use Velero with CSI snapshots for Kubernetes-level backups.
• Encryption: vSphere VM Encryption and Cinder volume encryption both require a Key Management Server (KMS). Deploy a KMS (e.g., Barbican for OpenStack, HyTrust for vSphere) before enabling encryption.
• Storage DRS (vSphere): If using multiple VMFS datastores, enable Storage DRS to balance capacity and I/O across datastores automatically.
• Ceph tuning: For Ceph backends, set osd_op_queue_cut_off=high and tune osd_recovery_max_active to limit recovery I/O impact on production workloads.