Reference Architecture: Virtual Networking
Purpose: For platform engineers, provides virtual switch configuration, distributed port groups, and overlay network specifications.
Overview
Virtual networking bridges the physical network (see Physical Network) to the Kubernetes VMs. This layer handles VLAN trunking, traffic shaping, and overlay networks. Configuration differs between vSphere (vDS/NSX) and OpenStack (Neutron/OVS/OVN).
vSphere Virtual Networking
vSphere Distributed Switch (vDS)
Use a vSphere Distributed Switch for consistent network policy across all ESXi hosts. Standard vSwitches are acceptable for single-host labs but not for production.
| Setting | Value |
|---|---|
| vDS Version | 7.0 or 8.0 (match vCenter version) |
| MTU | 9000 (for vMotion/storage port groups) |
| Discovery Protocol | LLDP (both listen and advertise) |
| Network I/O Control (NIOC) | Enabled |
| Health Check | Enabled (VLAN and MTU checks) |
Port Groups
| Port Group | VLAN | Teaming Policy | Active Uplinks | MTU | Traffic Type |
|---|---|---|---|---|---|
| Management | 10 | Active/Standby | uplink1 (active), uplink2 (standby) | 1500 | ESXi mgmt, vCenter |
| vMotion | 20 | Active/Standby | uplink2 (active), uplink1 (standby) | 9000 | Live migration |
| Storage | 30 | Route based on IP hash | uplink1, uplink2 | 9000 | vSAN, NFS, iSCSI |
| VM Network | 40 | Route based on originating port | uplink1, uplink2 | 1500 | Kubernetes node IPs |
Stagger active uplinks across port groups to distribute traffic across both physical NICs. Management uses uplink1 active; vMotion uses uplink2 active.
Network I/O Control (NIOC) Shares
NIOC prevents a single traffic type from saturating the physical NICs.
| Traffic Type | Shares | Reservation |
|---|---|---|
| Management | High | 1 Gbps |
| vMotion | High | 5 Gbps |
| vSAN | High | 10 Gbps |
| VM Traffic | Normal | None |
| NFS | Normal | None |
Security Policies (per Port Group)
| Policy | Management | vMotion | Storage | VM Network |
|---|---|---|---|---|
| Promiscuous Mode | Reject | Reject | Reject | Reject |
| MAC Address Changes | Reject | Reject | Reject | Accept |
| Forged Transmits | Reject | Reject | Reject | Accept |
Set MAC Address Changes and Forged Transmits to "Accept" on the VM Network port group. Kubernetes CNI plugins (Calico, Cilium) require this for pod networking when using non-overlay modes.
VMware NSX (Optional)
NSX provides micro-segmentation, distributed firewalling, and overlay networking at the hypervisor level. Use NSX when:
- Multi-tenant isolation is required between Kubernetes clusters on the same hosts.
- Distributed firewall rules must be enforced below the VM OS level.
- The organization already operates NSX for other workloads.
NSX is not required for standard openCenter deployments. Calico or Cilium inside Kubernetes handles pod-level network policy.
| NSX Component | Purpose |
|---|---|
| NSX Manager | Central management (3-node cluster for HA) |
| Transport Zones | Overlay (GENEVE) and VLAN zones |
| Segments | Replace port groups for VM connectivity |
| Distributed Firewall | Micro-segmentation rules |
| Tier-0/Tier-1 Gateways | North-south routing |
OpenStack Virtual Networking (Neutron)
Network Backend
| Backend | Use Case | Performance |
|---|---|---|
| Open vSwitch (OVS) | Standard deployments | Good; mature and well-tested |
| OVN (Open Virtual Network) | New deployments, scale | Better; distributed control plane |
| SR-IOV | High-performance workloads | Best; hardware offload, bypasses hypervisor |
OVN is the recommended backend for new OpenStack deployments. It eliminates the central Neutron L3/DHCP agents and distributes routing to each compute node.
Network Topology
| Network | Type | CIDR (example) | Purpose |
|---|---|---|---|
| provider-mgmt | Flat / VLAN 10 | 10.0.10.0/24 | OpenStack API, node management |
| provider-external | Flat / VLAN 50 | 203.0.113.0/24 | Floating IPs, external access |
| k8s-internal | VXLAN (tenant) | 10.100.0.0/16 | Kubernetes node network |
| k8s-pod | VXLAN (tenant) | 10.200.0.0/16 | Pod overlay (Calico manages) |
Neutron Configuration
| Setting | Value | Reason |
|---|---|---|
| MTU (tenant networks) | 1450 | 1500 minus VXLAN overhead (50 bytes) |
| MTU (provider networks) | 1500 or 9000 | Match physical network |
| Security Groups | Enabled | Required for node-level firewall rules |
| Port Security | Enabled (default) | Disable per-port for Calico BGP if needed |
| Allowed Address Pairs | Configure for pod CIDRs | Required for Calico non-overlay mode |
| DNS | Neutron DNS integration | Resolves instance hostnames |
Floating IPs
Allocate floating IPs for:
| Purpose | Count | Notes |
|---|---|---|
| Kubernetes API (control plane VIP) | 1 | Accessed by kubectl, CI/CD |
| Ingress controller / MetalLB | 2–5 | External service exposure |
| Bastion host | 1 | SSH jump host |
Kubernetes Overlay Network (Calico)
Calico runs inside the Kubernetes cluster and manages pod-to-pod networking. The virtual network layer must allow Calico traffic to pass.
Calico Modes
| Mode | Encapsulation | Performance | Requirements |
|---|---|---|---|
| VXLAN (default) | VXLAN | Good | No special network config |
| IP-in-IP | IPIP | Good | IP protocol 4 allowed |
| BGP (no overlay) | None | Best | BGP peering with ToR switches or Neutron |
VXLAN is the default and works on all platforms without network changes. BGP mode provides better performance but requires coordination with the physical or virtual network team.
Firewall Rules for Calico
If firewalls exist between Kubernetes nodes, allow:
| Protocol | Port | Purpose |
|---|---|---|
| UDP | 4789 | VXLAN encapsulation |
| TCP | 179 | BGP (if using BGP mode) |
| IP Protocol 4 | — | IPIP (if using IP-in-IP mode) |
| TCP | 5473 | Calico Typha |
| TCP | 9091 | Calico Felix metrics |
Considerations
- MTU consistency: Verify MTU end-to-end from pod → VM NIC → virtual switch → physical switch → destination. A single MTU mismatch causes silent packet drops or fragmentation. Use
ping -M do -s 8972 <destination>to test jumbo frame path (8972 + 28 byte header = 9000). - VLAN pruning: Only trunk the VLANs each port group needs. Do not trunk all VLANs to all hosts — this increases the broadcast domain and attack surface.
- Monitoring: Enable NetFlow/sFlow on the vDS or OVS to capture traffic statistics. Feed into the Grafana stack for network visibility.
- IPv6: openCenter clusters use IPv4. If the underlying virtual network is dual-stack, ensure IPv4 connectivity is not affected by IPv6 router advertisements.
- NSX vs. Calico: NSX operates at the hypervisor level (below the VM). Calico operates inside the Kubernetes cluster (above the VM). They are complementary, not competing. Use NSX for VM-level isolation and Calico for pod-level policy.