Skip to content

Dispatch backplane

When Ampora runs as more than one instance, the instances need to coordinate on three things:

  1. Agent session ownership — which instance's WebSocket holds an agent.
  2. Cross-instance dispatch — when an operator action on instance A needs to reach an agent owned by instance C.
  3. Live UI updates — events propagated to every connected operator's browser regardless of which instance their SignalR circuit is on.

All three ride the dispatch backplane. Three implementations are shipped.

Backplane When to pick it Extra deps
InProcess Single-instance deployments none
Postgres Multi-instance, no extra services the existing PostgreSQL
Redis Multi-instance, lower latency, separation of concerns Redis 6+

Choosing

  • InProcess is the default. It is correct and fast for one instance. It is also a no-op — every dispatch becomes a local function call.
  • Postgres uses LISTEN/NOTIFY + a session-ownership table. It is the cleanest path to HA without bringing Redis into your stack: you already need Postgres. Latency is in the low milliseconds for a healthy Postgres.
  • Redis uses pub/sub. Pick this when (a) you already operate Redis for other reasons, or (b) your Postgres is intentionally a write-cold managed instance and you do not want the extra notification load on it.

You can switch back and forth between Postgres and Redis without data loss — the durable state lives in PostgreSQL either way.

Configuration

Dispatch:
  Backplane: Postgres            # InProcess | Postgres | Redis
  RedisConnectionString: ""      # required when Backplane = Redis
  OwnershipTtlSeconds: 60
  LeaderLeaseSeconds: 30

Or as env vars:

Dispatch__Backplane=Postgres
Dispatch__RedisConnectionString=redis://redis.acme.svc.cluster.local:6379,abortConnect=false
Dispatch__OwnershipTtlSeconds=60
Dispatch__LeaderLeaseSeconds=30

Session ownership

When an agent connects, its handler instance INSERTs a row in agent_session_ownership with a fencing token (a monotonically increasing bigint) and a TTL. The instance renews the lease while the session stays open. On graceful shutdown, the row is DELETED and the agent reconnects to whichever instance the load balancer routes it to. On crash, the row's TTL expires and another instance can re-acquire on the next connect.

The fencing token defends against split-brain: if instance A is network-partitioned away but still believes it owns the session, and instance B re-acquires (with a higher token), A's writes carry the older token and are rejected at the consumer side.

Cross-instance dispatch

An operator clicks "Push config v3 to agent X" on instance A. The flow:

  1. A asks the ownership table: who owns agent X?
  2. Answer: instance C.
  3. A publishes a DispatchEnvelope (with payload reference, fencing token, target agent ID) on the configured transport (LISTEN/NOTIFY topic or Redis channel).
  4. C's listener receives the envelope, fetches the payload from the database, validates the fencing token, and pushes the ServerToAgent frame on its WebSocket.
  5. The agent applies the config and reports back through C, which propagates the result via the live-update bus to every connected operator UI.

Latency from "operator clicks" to "agent receives frame" stays under about 2 seconds in healthy clusters. The ADR has the wire-level details: ADR-028.

Live UI updates

Two transport adapters mirror the dispatch backplane:

  • PostgresLiveUpdateListener — listens on a different LISTEN topic for UI events.
  • RedisLiveUpdateListener — same idea over Redis pub/sub.

A BackplaneLiveUpdateBus wraps the in-process bus and adds a peer-broadcast layer with a 200 ms coalescing window. Sender-instance filtering prevents the broadcasting instance from echoing its own events back into its own UI.

The result: operator A on instance #1 starts a rollout, operator B on instance #3 watches the progress bar advance live without refreshing.

Leader-elected work

A small set of background jobs run on a single instance at a time:

  • Drift reconciler (tags out-of-date agents).
  • Audit retention sweep.
  • GitOps sync.
  • Dynamic group reconciler.
  • Rollout health-gate ticker.

Each holds a named lease in the database (leader_leases table) with a TTL. If the holding instance crashes, another instance grabs the lease on the next tick. The fencing token guarantees the kicked-out leader cannot re-acquire mid-cycle.

Sticky sessions are still required

The dispatch backplane handles server-to-agent routing. The operator browser → server path is a Blazor Server SignalR circuit that must stick to one instance for the life of the session. Configure your reverse proxy with cookie-based affinity. The High availability page has ready-made snippets for Nginx Ingress, Traefik, and Azure Application Gateway.