Reference Architecture: Physical Network
Purpose: For platform engineers, provides physical network specifications including top-of-rack switching, spine-leaf topology, VLAN design, and MTU requirements.
Overview
openCenter clusters require a reliable, low-latency physical network. The reference architecture uses a spine-leaf topology with redundant top-of-rack (ToR) switches. This design provides predictable east-west bandwidth between hypervisor hosts, which is critical for etcd consensus, vMotion, and storage replication traffic.
Network Topology
Spine-Leaf (Recommended)
Use spine-leaf for deployments of two or more racks. Every leaf switch connects to every spine switch, providing equal-cost multipath (ECMP) between any two hosts.
Single-Rack (Small Deployments)
For a single rack, use a pair of ToR switches in an MLAG/VPC configuration. This avoids the cost of spine switches while still providing redundancy.
Switch Specifications
Spine Switches
| Specification | Minimum | Recommended |
|---|---|---|
| Port Count | 32× 100 GbE QSFP28 | 64× 100 GbE QSFP28 |
| Switching Capacity | 6.4 Tbps | 12.8 Tbps |
| Latency | < 1 µs port-to-port | < 500 ns port-to-port |
| Routing | BGP, OSPF, ECMP | BGP, OSPF, ECMP, EVPN |
| Example Models | Arista 7050X3, Cisco Nexus 9336C-FX2 | Arista 7060X4, Cisco Nexus 9364C-GX |
Leaf / ToR Switches
| Specification | Minimum | Recommended |
|---|---|---|
| Downlink Ports | 48× 10/25 GbE SFP28 | 48× 25 GbE SFP28 |
| Uplink Ports | 4× 100 GbE QSFP28 | 8× 100 GbE QSFP28 |
| Switching Capacity | 3.6 Tbps | 6.4 Tbps |
| MLAG/VPC | Required | Required |
| Example Models | Arista 7050SX3-48YC12, Dell S5248F-ON | Arista 7060SX2-48YC6, Cisco Nexus 93180YC-FX3 |
Out-of-Band Management Switch
| Specification | Value |
|---|---|
| Ports | 24–48× 1 GbE RJ45 |
| Purpose | BMC/IPMI/iLO/iDRAC traffic only |
| Example Models | Arista 720XP-24ZY2, Cisco Catalyst 9200L |
VLAN Design
Separate traffic types into distinct VLANs. This provides security isolation and allows per-VLAN QoS policies.
| VLAN ID | Name | Subnet (example) | MTU | Purpose |
|---|---|---|---|---|
| 10 | Management | 10.0.10.0/24 | 1500 | ESXi/KVM management, vCenter, BMC |
| 20 | vMotion | 10.0.20.0/24 | 9000 | Live migration traffic |
| 30 | Storage | 10.0.30.0/24 | 9000 | vSAN, NFS, iSCSI |
| 40 | VM Network | 10.0.40.0/24 | 1500 | Kubernetes node IPs |
| 50 | Kubernetes Pod | 10.0.50.0/16 | 1500 | Pod overlay (Calico VXLAN/BGP) |
| 100 | OOB Management | 10.0.100.0/24 | 1500 | IPMI/iLO/iDRAC (isolated) |
Adjust VLAN IDs and subnets to match your existing network standards. The key requirement is traffic separation, not specific VLAN numbers.
MTU Configuration
| Traffic Type | MTU | Justification |
|---|---|---|
| Management | 1500 | Standard; no benefit from jumbo frames |
| vMotion | 9000 | Reduces CPU overhead for large memory transfers |
| Storage (vSAN/iSCSI/NFS) | 9000 | Required for storage performance; test end-to-end |
| VM / Kubernetes Node | 1500 | Standard; overlay adds headers |
| Kubernetes Pod (VXLAN) | 1450 | 1500 minus 50-byte VXLAN overhead |
Jumbo frames (MTU 9000) must be configured end-to-end: server NIC → ToR switch → spine switch → destination ToR → destination NIC. A single link at MTU 1500 in the path silently fragments traffic and degrades performance.
Link Aggregation and Redundancy
- Each server connects to two ToR switches (one link per switch) using LACP (802.3ad) or active-standby bonding.
- ToR switch pairs run MLAG (Arista), VPC (Cisco), or VLT (Dell) for multi-chassis link aggregation.
- Spine switches use ECMP with BGP or OSPF for equal-cost load balancing across all leaf uplinks.
Server NIC Assignment (4-port example)
| Port | Switch | Traffic |
|---|---|---|
| NIC0 (25 GbE) | Leaf-01 | Management + VM Network (LACP bond0) |
| NIC1 (25 GbE) | Leaf-02 | Management + VM Network (LACP bond0) |
| NIC2 (25 GbE) | Leaf-01 | vMotion + Storage (LACP bond1) |
| NIC3 (25 GbE) | Leaf-02 | vMotion + Storage (LACP bond1) |
For two-port servers, trunk all VLANs on a single LACP bond. This reduces isolation but meets the minimum requirement.
Bandwidth Requirements
These values come from Capacity & Sizing:
| Traffic Type | Minimum | Recommended |
|---|---|---|
| Control Plane (etcd, API server) | 1 Gbps | 10 Gbps |
| Worker Node (pod-to-pod) | 1 Gbps | 10 Gbps |
| Storage Network | 10 Gbps | 25 Gbps |
| vMotion | 10 Gbps | 25 Gbps |
Routing and Gateway
- Use a dedicated router or Layer 3 switch as the default gateway for each VLAN.
- If using spine-leaf with BGP, configure each leaf as a Layer 3 gateway for its local VLANs (distributed routing).
- Firewall the OOB management VLAN from all other networks. BMC interfaces should not be reachable from the VM network.
Considerations
- Cable budget: 25 GbE SFP28 DAC cables work up to 5 m (within a rack). Use active optical cables (AOC) or transceivers for inter-rack runs. See Cabling Standards.
- QoS: If sharing the network with non-Kubernetes traffic, configure DSCP markings and priority queuing for storage and vMotion VLANs.
- Spanning Tree: Spine-leaf with BGP/ECMP eliminates STP. If using a traditional two-tier design, enable RSTP and configure root bridges explicitly.
- IPv6: openCenter supports IPv4-only clusters. IPv6 or dual-stack requires additional planning outside this reference architecture.
- Network monitoring: Deploy SNMP polling and sFlow/NetFlow collection on all switches. Feed into the Prometheus/Grafana stack via SNMP exporter.