Skip to main content

ADR-002: OpenStack CCM Serial API Calls Default

Purpose: For platform engineers and operators, explains why provider-requires-serial-api-calls: true is the default for all OpenStack CCM deployments.

Status​

Accepted

Context​

The OpenStack Cloud Controller Manager (CCM) integrates Kubernetes with OpenStack infrastructure for load balancers, storage volumes, and node lifecycle. On Rackspace OpenStack, the load balancer API (Octavia-based) exhibits race conditions under concurrent API calls, causing:

  • Load balancer creation failures
  • Inconsistent load balancer state (HTTP 409 Conflict, HTTP 500)
  • Service reconciliation errors

Analysis of three production clusters showed inconsistent configuration. Only the Rackspace-GDO cluster included the critical provider-requires-serial-api-calls: true setting, and it operated without load balancer API errors. The other two clusters experienced intermittent failures.

The setting controls whether load balancer API operations are serialized:

  • false (upstream default): CCM makes concurrent API calls
  • true: CCM serializes calls, ensuring one operation completes before the next starts

Rackspace OpenStack requires serialization because of state machine constraints in Octavia, pessimistic database locking, and API gateway behavior that rejects concurrent requests to the same resource.

Decision​

Make provider-requires-serial-api-calls: true the default for all OpenStack CCM deployments in openCenter. This applies to openCenter-gitops-base Helm values, openCenter-cli generated configurations, and all customer overlays.

Rationale​

  1. Serial API calls prevent race conditions without meaningful performance impact. Load balancer operations (service creation/deletion) are infrequent, so serialization overhead is negligible.
  2. The Rackspace-GDO cluster demonstrates this configuration works reliably in production.
  3. A conservative default avoids API errors. Operators who need concurrency can override.

Configuration​

Applied in the OpenStack CCM Helm values:

cloudConfig:
  loadBalancer:
    floating-network-id: <network-id>
    subnet-id: <subnet-id>
    provider-requires-serial-api-calls: true

Consequences​

Positive:

  • Eliminates load balancer API race conditions on Rackspace OpenStack
  • Consistent behavior across all customer clusters
  • No operator intervention needed for new deployments

Trade-offs:

  • Concurrent load balancer operations are queued, adding latency during bulk service creation
  • Operators on non-Rackspace OpenStack may not need this setting, but it causes no harm