Usage#

Use the local-path StorageClass in your PVC. Note that pods using this storage are bound to the node where the data resides.

2) Tier 2: Rook Ceph (Shared & Resilient)#

Rook turns Ceph into a cloud-native storage service. It provides block (RBD), file (CephFS), and object (S3) storage with replication.

2-1. Install Rook Operator#

2-2. Deploy Ceph Cluster#

Apply a production-ready cluster configuration. For a 3-node homelab, ensure allowMultiplePerHost: true if you have fewer nodes than replicas.

cluster.yaml (simplified):

Tip: If you have fewer than 3 nodes, set allowMultiplePerHost: true and reduce mon.count to match your node count.

2-3. Define StorageClasses#

Create StorageClasses for Block and File storage to enable dynamic provisioning. Make sure the pools and filesystem names exist in your Ceph cluster.

Ceph Block (RBD):

Ceph File (CephFS):

2-4. Rook Ceph Trade-offs (Pros and Cons)#

Rook gives you real HA storage in Kubernetes, but you are operating a full distributed storage system on top of your cluster. In a homelab, that trade-off is often the limiting factor.

Personal incident I tried to reduce OSD resource requests because they were too heavy for my nodes. I tweaked the Rook operator after a Helm chart deploy, and the OSD pods all failed. The cluster went into a bad state and I could not recover the data. I lost ~1 TB. Treat OSD tuning as a last resort and test changes on a throwaway cluster first.

Related incidents reported by others

• ARM64 OSD crash on v18.2.4 - OSD crashes reported on ARM64.
• OSD disk preparation crashes if the disks are encrypted - OSD preparation failures with encrypted disks.
• Some OSDs cannot re-authenticate to cluster after cluster failure - OSDs stuck after a cluster failure event.
• Ceph Tentacle v20.2.0 may have data loss with CephFS when OSD readAffinity is enabled - data loss risk in CephFS with readAffinity.
• CephFS: Data loss when readAffinity is enabled on Ceph Tentacle v20.2.0 - corroborating report from Ceph-CSI.

When it works best

• Dedicated storage nodes or clear failure domains.
• Stable power (UPS) and decent network (10GbE if possible).
• You accept that storage availability is tied to cluster availability.

3) Storage Mapping to Workloads#

• Local Path: Single-node performance (e.g., CI/CD runners, temp DBs).
• Rook Ceph (RBD): Shared block storage for standard apps (e.g., Ghost, Grafana).
• Rook Ceph (CephFS): Shared file system for multi-writer apps (e.g., shared media).

4) Verification#

Check the status of your Ceph cluster:

Wrap-up#

By combining Local Path and Rook Ceph, this homelab achieves a balance between raw performance and high availability. Part 4 will cover operational tooling like Velero and K9s.