Concepts — Pragmatic.ControlPlane Architecture

Deep dive into the SignalR-based coordination layer: hub and client architecture, health aggregation, command dispatch, leader election, and graceful degradation.

---

Overview

The control plane is a coordination layer for distributed Pragmatic hosts. It solves four problems:

1. Migration safety — Only one instance runs migrations at a time
2. Visibility — Which hosts are running, healthy, or stuck
3. Orchestration — Drain, maintenance mode, deploy coordination across instances
4. Configuration push — Runtime config updates without restart

                    ┌─────────────────────────────┐
                    │   ControlPlaneHub (SignalR)  │
                    │   ┌───────────────────────┐  │
                    │   │ ControlPlaneHubRegistry│  │
                    │   │ CommandAuditLog        │  │
                    │   │ StaleHostEviction      │  │
                    │   │ ControlPlaneAuthFilter │  │
                    │   └───────────────────────┘  │
                    └──────┬────────┬────────┬─────┘
                           │        │        │
                    ┌──────▼──┐ ┌──▼────┐ ┌─▼──────┐
                    │ Host A  │ │Host B │ │ Host C  │
                    │ (Client)│ │(Client│ │(Client) │
                    └─────────┘ └───────┘ └─────────┘

---

1. Packages and Boundaries

Package	Responsibility	Key Types
Pragmatic.Abstractions	Interfaces only	IControlPlane, IHostIdentity, IHostStatus, IHostCommandHandler<T>, IHostHealthContributor
Pragmatic.Composition.Host	Local implementations	LocalHostIdentity, LocalHostStatus, NoOpControlPlane, HostCommandDispatcher, HostHealthAggregator
Pragmatic.ControlPlane.SignalR	Hub (server side)	ControlPlaneHub, ControlPlaneHubRegistry, CommandAuditLog, ControlPlaneAuthFilter, StaleHostEvictionService
Pragmatic.ControlPlane.Client	Client (each instance)	SignalRControlPlane, ControlPlaneConnectionService, ControlPlaneClientOptions
Pragmatic.MultiTenancy.Persistence	Tenant orchestration	TenantMigrationOrchestrator, TenantConnectionStringProvider, ITenantDatabaseProvisioner

The abstractions live in Pragmatic.Abstractions so any module can depend on IHostStatus or IControlPlane without pulling in SignalR or ASP.NET Core.

---

2. Host Identity and Lifecycle

IHostIdentity

Every Pragmatic host has a singleton identity, registered automatically at startup:

public interface IHostIdentity
{
    string HostId { get; }           // Guid7, regenerated each startup
    string HostName { get; }         // From SG topology (e.g. "Showcase.Host")
    HostType HostType { get; }       // Monolith, Distributed, Standalone
    DateTimeOffset StartedAt { get; }
}

LocalHostIdentity (in Composition.Host) derives HostName from the assembly name and HostId from Guid.CreateVersion7().

IHostStatus

Mutable runtime state with a defined lifecycle:

Starting → Ready → Serving → Draining → Stopping → Stopped
                     ↕
                Maintenance
                     ↕
                Migrating

public interface IHostStatus
{
    HostState State { get; }
    string? StateReason { get; }
    DateTimeOffset StateChangedAt { get; }
    MigrationStatus? MigrationStatus { get; }

    void TransitionTo(HostState newState, string? reason = null);
    void UpdateMigrationProgress(MigrationStatus status);
}

LocalHostStatus is the concrete implementation. State transitions are thread-safe. The StateChangedAt timestamp is updated on every transition.

HostInfo

The wire format sent over SignalR:

public sealed record HostInfo
{
    public string HostId { get; init; }
    public string HostName { get; init; }
    public HostType HostType { get; init; }
    public HostState State { get; init; }
    public string? StateReason { get; init; }
    public MigrationStatus? MigrationStatus { get; init; }
    public DateTimeOffset LastHeartbeat { get; init; }
    public DateTimeOffset StartedAt { get; init; }
}

---

3. SignalR Hub

ControlPlaneHub

A strongly-typed SignalR hub (Hub<IControlPlaneHubClient>) that provides:

Method	Direction	Description
Register(HostInfo)	Client to Hub	Register on connect
Heartbeat(hostId, state, reason, migrationStatus)	Client to Hub	Periodic status update
SendCommand(targetHostId, commandType, commandJson)	Client to Hub	Routed command
PushConfigChange(tenantId, key, oldValue, newValue)	Hub to All	Config broadcast
AnnounceDeployment(version, duration, requiresMaintenance)	Hub to All	Deploy notification
AnnounceDeploymentComplete(version, success)	Hub to All	Deploy completion
GetAllHosts()	Client to Hub	Initial roster
ClaimMigrationLeadership(databaseName, hostId)	Client to Hub	Migration lock
ReleaseMigrationLeadership(databaseName, hostId)	Client to Hub	Migration unlock

ControlPlaneHubRegistry

Thread-safe ConcurrentDictionary mapping hostId to HostConnectionInfo:

public sealed class ControlPlaneHubRegistry
{
    // Host tracking
    void Register(string connectionId, HostInfo hostInfo);
    (string HostId, string HostName)? RemoveByConnectionId(string connectionId);
    void UpdateStatus(string hostId, HostState state, string? reason, MigrationStatus?);
    IReadOnlyList<HostInfo> GetAllHosts();
    HostInfo? GetHost(string hostId);

    // Stale detection
    IReadOnlyList<string> GetStaleHosts(TimeSpan timeout);
    HostConnectionInfo? Remove(string hostId);

    // Migration coordination
    bool TryClaimMigrationLeadership(string databaseName, string hostId);
    bool ReleaseMigrationLeadership(string databaseName, string hostId);
    IReadOnlyList<string> CleanupMigrationLeadership(string hostId);
}

Migration leadership is tracked per database name. If a host disconnects, CleanupMigrationLeadership releases all its locks.

Stale Host Eviction

StaleHostEvictionService runs as a BackgroundService. It periodically checks GetStaleHosts(timeout) and removes hosts that have not sent a heartbeat within the configured StaleEvictionInterval (default: 2 minutes).

When a host is evicted:

1. Its HostConnectionInfo is removed from the registry
2. An OnHostDisconnected event is broadcast to all other clients
3. Any migration leadership held by the host is released

Authentication

ControlPlaneAuthFilter implements IHubFilter. It checks the X-Control-Plane-Key header on every hub invocation:

• If no API key is configured on the hub, all connections are accepted
• If an API key is configured, connections without a matching key are rejected
• Scoped permissions (read-only, command, admin) can be configured per key

Command Audit Log

Every command dispatched through SendCommand is recorded in CommandAuditLog:

public sealed record CommandAuditEntry
{
    public DateTimeOffset Timestamp { get; init; }
    public string SenderHostId { get; init; }
    public string TargetHostId { get; init; }
    public string CommandType { get; init; }
    public string CommandPayload { get; init; }
    public bool Success { get; init; }
    public string? ErrorMessage { get; init; }
}

The audit log is queryable via the REST endpoint GET /_pragmatic/control-plane/audit.

---

4. SignalR Client

SignalRControlPlane

Implements IControlPlane by connecting to a remote hub:

public sealed class SignalRControlPlane : IControlPlane, IAsyncDisposable
{
    bool IsConnected { get; }

    Task ReportStatusAsync(CancellationToken ct);
    Task<IReadOnlyList<HostInfo>> GetAllHostsAsync(CancellationToken ct);
    IAsyncEnumerable<ControlPlaneEvent> StreamEventsAsync(CancellationToken ct);
    Task<ControlPlaneError?> SendCommandAsync(string targetHostId, HostCommand command, CancellationToken ct);
    Task BroadcastEventAsync(ControlPlaneEvent evt, CancellationToken ct);

    Task ConnectAsync(CancellationToken ct);
    Task DisconnectAsync(CancellationToken ct);
}

Key behaviors:

• Auto-reconnect: Exponential backoff with MaxReconnectAttempts. After exhausting retries, degrades to NoOp.
• Lazy connection: Does not connect at construction. ControlPlaneConnectionService triggers connection after ApplicationStarted.
• Graceful degradation: If the hub is unreachable, IsConnected returns false and all operations become no-ops.
• Re-registration: On reconnect, the client re-registers its HostInfo with the hub.

Event Channel

Events from the hub are pushed into a Channel<ControlPlaneEvent>. Consumers read via StreamEventsAsync():

Hub Callback	Event Type
OnHostStateChanged	HostStateChangedEvent
OnMigrationProgress	MigrationProgressChangedEvent
OnCommand	Dispatched to IHostCommandDispatcher
OnConfigChanged	ConfigChangedEvent
OnDeploymentAnnounced	DeploymentAnnouncedEvent
OnDeploymentCompleted	DeploymentCompletedEvent

Connection Lifecycle

ControlPlaneConnectionService manages the SignalR connection:

Host.StartAsync()
  → Register ApplicationStarted callback
    → ConnectAsync() (after Kestrel is listening)
      → _hub.StartAsync()
      → _hub.InvokeAsync("Register", hostInfo)
      → Start heartbeat timer

Host.StopAsync()
  → Stop heartbeat timer
  → TransitionTo(Stopped)
  → ReportStatusAsync() (final heartbeat)
  → DisconnectAsync()

The ApplicationStarted callback is critical for embedded hub mode: it ensures the SignalR hub is listening before the client tries to connect.

---

5. Health Contributors

IHostHealthContributor

Modules contribute health status without coupling to the control plane:

public interface IHostHealthContributor
{
    string Name { get; }
    Task<HealthContribution> CheckAsync(CancellationToken ct);
}

Health Flow

Health flows in two modes:

1. Push — Included in heartbeat payloads sent to the hub. The IHostHealthAggregator collects contributions from all registered IHostHealthContributor instances.

2. Pull — Aggregated on-demand via IHostStatus.GetHealthAsync(). Used by the local health check endpoint.

The hub aggregates health from all connected hosts. The REST endpoint GET /_pragmatic/control-plane/hosts includes health status for each host.

ContributorHealthStatus

public enum ContributorHealthStatus
{
    Healthy,
    Degraded,
    Unhealthy
}

---

6. Command Dispatch

Architecture

Sender Host                  Hub                    Target Host
    │                         │                         │
    │ SendCommand(target,     │                         │
    │   "DrainCommand", json) │                         │
    │ ─────────────────────▶  │                         │
    │                         │ OnCommand(sender,       │
    │                         │   "DrainCommand", json) │
    │                         │ ────────────────────▶   │
    │                         │                         │
    │                         │               IHostCommandDispatcher
    │                         │               .DispatchAsync()
    │                         │                    │
    │                         │               IHostCommandHandler<DrainCommand>
    │                         │               .HandleAsync()

HostCommandDispatcher

HostCommandDispatcher (in Composition.Host) receives the command type name and JSON payload, deserializes it, and resolves the appropriate IHostCommandHandler<T> from DI.

Built-in Commands

Command	Handler	Effect
MaintenanceCommand	MaintenanceCommandHandler	Toggles HostState.Maintenance via IMaintenance
DrainCommand	DrainCommandHandler	Transitions to HostState.Draining, stops accepting new requests

Custom Commands

Define a record inheriting from HostCommand:

public sealed record ClearCacheCommand(string CacheName) : HostCommand;

Implement the handler:

public sealed class ClearCacheHandler(ICacheManager cacheManager)
    : IHostCommandHandler<ClearCacheCommand>
{
    public async Task HandleAsync(ClearCacheCommand command, CancellationToken ct)
    {
        await cacheManager.ClearAsync(command.CacheName, ct);
    }
}

Register in DI. The dispatcher discovers it automatically.

---

7. Migration Coordination via Hub

When the control plane is active, migration leader election can be coordinated through the hub instead of (or in addition to) DatabaseLeaderElection:

Host A: ClaimMigrationLeadership("AppDb", hostA) → true  (leader)
Host B: ClaimMigrationLeadership("AppDb", hostB) → false (follower)
Host C: ClaimMigrationLeadership("AppDb", hostC) → false (follower)

Host A runs migrations...

Host A: ReleaseMigrationLeadership("AppDb", hostA)
Hub broadcasts: OnHostStateChanged(hostA, Migrating, Ready)

The ControlPlaneHubRegistry tracks migration leaders per database. If a host disconnects, CleanupMigrationLeadership releases all its locks and notifies other hosts.

---

8. Graceful Degradation

The control plane is entirely opt-in. Every interface has a NoOp default:

Interface	NoOp Behavior
IHostIdentity	LocalHostIdentity — hostname + PID
IHostStatus	LocalHostStatus — always allows transitions, local only
IControlPlane	NoOpControlPlane — returns self only, commands silently ignored
ILeaderElection	Always grants leadership (single instance assumption)
IHostCommandHandler<T>	Not registered — commands ignored

Degradation cascade:

Hub reachable     → Full coordination (heartbeat, commands, events)
Hub unreachable   → Client degrades to NoOp (host keeps serving)
No UseControlPlane→ All NoOp defaults (zero overhead)

The client detects hub unavailability and logs: Control plane connection closed permanently, operating in degraded mode. The host continues serving without coordination.

---

9. Security

API Key Authentication

The hub validates an API key on every SignalR invocation:

Client → X-Control-Plane-Key: my-cluster-secret → Hub

The ControlPlaneAuthFilter (IHubFilter) rejects connections without a valid key.

Transport Security

SignalR runs over HTTPS. Use standard ASP.NET Core TLS configuration:

{
  "Kestrel": {
    "Endpoints": {
      "ControlPlane": {
        "Url": "https://0.0.0.0:5100",
        "Certificate": {
          "Path": "/certs/control-plane.pfx"
        }
      }
    }
  }
}

Audit Trail

All commands dispatched through the hub are recorded in CommandAuditLog with sender, target, type, payload, success/failure, and timestamp.

---

10. Observability

OpenTelemetry

ActivitySource: Pragmatic.ControlPlane

Spans for: command dispatch, leader election, heartbeat, migration coordination.

Metrics

Metric	Type	Description
pragmatic.controlplane.heartbeats	Counter	Heartbeats sent
pragmatic.controlplane.commands_dispatched	Counter	Commands routed through hub
pragmatic.controlplane.leader_elections	Counter	Leader election attempts
pragmatic.controlplane.reconnections	Counter	Client reconnection attempts
pragmatic.controlplane.hosts_registered	Gauge	Currently registered hosts

Health Check

ControlPlaneHealthCheck (in Composition.Host) reports the control plane connection status to ASP.NET Core health checks:

• Connected to hub: Healthy
• Reconnecting: Degraded
• Disconnected: Unhealthy (or Degraded if degradation is expected)

---

Architecture Diagram

Pragmatic.Abstractions/ControlPlane/
├── IControlPlane                    # Core coordination interface
├── IHostIdentity                    # Host identity (read-only)
├── IHostStatus                      # Mutable runtime state
├── IHostCommandHandler<T>           # Command handling
├── IHostCommandDispatcher           # Command routing
├── IHostHealthContributor           # Health reporting
├── IHostHealthAggregator            # Health collection
├── HostCommand                      # Base command record
├── HostInfo                         # Wire format
├── HostState                        # State enum
├── HostType                         # Type enum
├── MigrationStatus                  # Migration progress
├── ControlPlaneEvent                # Event base
└── ControlPlaneError                # Error type

Pragmatic.Composition.Host/ControlPlane/
├── LocalHostIdentity                # Default identity
├── LocalHostStatus                  # Default status
├── NoOpControlPlane                 # Monolith fallback
├── HostCommandDispatcher            # DI-based command routing
├── HostHealthAggregator             # Collects IHostHealthContributor
├── HostStatusSyncService            # Syncs status to control plane
├── MaintenanceCommandHandler        # Built-in: toggle maintenance
├── DrainCommandHandler              # Built-in: graceful drain
└── ControlPlaneHealthCheck          # ASP.NET Core health check

Pragmatic.ControlPlane.SignalR/
├── ControlPlaneHub                  # SignalR hub
├── ControlPlaneHubRegistry          # Host registry + migration locks
├── ControlPlaneHubEndpoints         # REST fallback + DI registration
├── ControlPlaneHubOptions           # Hub configuration
├── ControlPlaneAuthFilter           # API key validation (IHubFilter)
├── ControlPlaneEndpointConfigurator # Maps hub in PragmaticApp pipeline
├── CommandAuditLog                  # Immutable command history
└── StaleHostEvictionService         # Background stale host cleanup

Pragmatic.ControlPlane.Client/
├── SignalRControlPlane              # IControlPlane implementation
├── ControlPlaneConnectionService    # Lifecycle (IHostedService)
└── ControlPlaneClientOptions        # Client configuration