Skip to main content

Aqueduct Operations

Overview​

Changing an Aqueduct stream provider or server stream namespace changes the routing identity shared by the backplane's runtime and gateway hosts. Changing the gateway broadcast namespace changes all-client routing among gateways. Treat either change as a coordinated maintenance operation across every participating host.

When this matters​

Use this page when changing StreamProviderName or ServerStreamNamespace across runtimes and gateways, changing AllClientsStreamNamespace among gateways, or recovering after a restart or failover that removed in-memory connections, groups, or server-directory state.

An API-only cutover that preserves the existing identities should follow the Aqueduct Runtime Composition Migration guidance. The maintenance procedure below is for an identity or provider change.

Prerequisites and assumptions​

  • Inventory the exact StreamProviderName and ServerStreamNamespace on every runtime and gateway that participates in the backplane. Inventory AllClientsStreamNamespace on every participating gateway.
  • Identify all message producers, SignalR gateways, Orleans runtimes, and clients that must move together.
  • Confirm the provider-specific procedure for preserving or recovering any durable stream or subscription metadata.
  • Treat Aqueduct connection, group-membership, and server-directory state as volatile in-memory state. Aqueduct does not persist that state through IGrainStorage.
  • Do not assume mixed runtime versions, mixed stream identities, a drain API, automatic replay, or zero-loss behavior; those behaviors are not established by the current implementation.

Keep StreamProviderName and ServerStreamNamespace unchanged and identical across participating runtimes and gateways for the ordinary runtime API cutover. Keep AllClientsStreamNamespace unchanged and identical among participating gateways. The broadcast namespace is gateway-owned; the runtime builder does not set or validate it. Configure the provider and server namespace through the runtime builder on runtimes and the corresponding gateway options on gateways. Configure the broadcast namespace only through gateway options, keeping one matching role-specific configuration set across participating hosts.

If an identity must change, schedule a maintenance window that stops message production, new traffic, and all participating gateways and runtimes before any host receives the new values. The whole participating backplane is the blast radius: old and new identities address different streams, so a partially updated fleet can partition delivery.

Procedure​

  1. Record the current provider and server-namespace values on every runtime and gateway, the broadcast namespace on every gateway, deployment versions, provider-owned durable metadata locations, and the clients or subscriptions that must be re-established.
  2. Stop application producers and stop accepting traffic that can create or use SignalR connections. Do not assume a drain operation exists; keep traffic stopped for the identity change.
  3. Stop participating gateway hosts first, then stop all participating runtime hosts. Verify that no old host remains able to publish or consume backplane messages.
  4. Apply one matching provider and server-namespace configuration set to every runtime and gateway deployment. Apply one matching broadcast namespace to every gateway. Keep the provider-owned durable metadata configuration explicit and unchanged unless the provider migration is intentional.
  5. Start all runtime hosts first. Wait for their normal Orleans health signal and verify that each runtime has the same provider and server-namespace values.
  6. Start the gateway hosts. Verify that each gateway has the same provider and server-namespace values as the runtimes and the same broadcast namespace as the other gateways. Allow each gateway to initialize its server and all-client stream subscriptions and register its heartbeat with the server directory.
  7. Re-establish client connections, group membership, and application subscriptions. Treat all previous in-memory membership as lost until these flows complete.
  8. Send controlled messages through the real connection, group, and broadcast paths. Verify receipt by the intended clients and confirm host health before resuming normal traffic.

Validation​

The change is ready for traffic only when all of the following are true:

  • Every participating runtime and gateway reports the same provider and server-namespace configuration.
  • Every participating gateway reports the same AllClientsStreamNamespace.
  • Runtime hosts report their normal Orleans started/healthy state.
  • Gateway logs show Orleans streams initialized for hub, Heartbeat manager started, and Orleans backplane initialized for each active hub/server; investigate any Heartbeat failed warning.
  • Clients have reconnected, groups have been rejoined, and application subscriptions have been recreated.
  • Controlled direct-connection, group, and broadcast messages traverse the intended paths and are observed by the intended clients.
  • No MSB201, MSB202, MSB206, or MSB207 composition diagnostics are present.
  • Provider-owned durable stream or subscription metadata has passed its provider-specific recovery check, when such metadata exists.

Failure modes and rollback​

Changing identities can make messages on the previous streams invisible to hosts using the new streams. Messages in flight during shutdown can be lost, and Aqueduct does not provide automatic replay. Stopping hosts also loses the in-memory client, group, and server-directory state; clients must reconnect, rejoin groups, and recreate subscriptions. This volatile membership is separate from any durable metadata owned by an external stream provider.

If validation fails while traffic is still stopped, roll back as one coordinated unit:

  1. Stop the gateways and runtimes using the new configuration.
  2. Restore the previous binary, provider and server-namespace configuration on every participating runtime and gateway, and restore the previous broadcast namespace on every gateway.
  3. Start the previous runtime set first and verify its Orleans health, provider, and server-namespace configuration.
  4. Start the previous gateway set and verify its provider and server namespace, broadcast namespace, stream initialization, and server registration.
  5. Reconnect clients, rejoin groups, recreate subscriptions, and repeat controlled direct, group, and broadcast message checks.
  6. Resume traffic only after the previous configuration and real message paths are healthy.

Recover provider-owned durable metadata through that provider's documented procedure. Do not treat volatile Aqueduct membership as recoverable storage, and do not resume traffic with old and new identity sets mixed.

Telemetry to watch​

Watch the host's Orleans health and stream-provider signals, plus Aqueduct's structured logs:

  • Orleans streams initialized for hub confirms gateway stream subscriptions completed.
  • Heartbeat manager started confirms gateway server registration began.
  • Orleans backplane initialized for hub confirms the gateway completed backplane setup.
  • Heartbeat failed for server is a warning requiring investigation before traffic resumes.
  • Runtime and gateway health checks should remain healthy after clients reconnect and controlled messages succeed.

These signals show initialization and liveness activity. They do not prove zero message loss or durable membership recovery.

Summary​

Keep the provider and server namespace stable and matching across runtimes and gateways for ordinary API-only changes, and keep the broadcast namespace stable and matching among gateways. For an intentional identity change, stop the whole participating fleet, apply one matching configuration set, start runtimes before gateways, rebuild volatile state, validate real message paths, and roll back the complete set while traffic remains stopped if validation fails.

Next Steps​