Common Mistakes — Pragmatic.ControlPlane

Eight pitfalls that developers hit when setting up distributed host coordination.

---

1. Hardcoded Hub URL Without Environment Awareness

Symptom: Clients connect to localhost:5100 in production, or to the production hub during local development.

Cause: The hub URL is hardcoded in Program.cs instead of coming from configuration.

// Wrong: hardcoded URL
builder.UseControlPlane(cp =>
{
    cp.WithHubUrl("https://localhost:5100/_pragmatic/control-plane");
});

// Correct: from configuration
builder.UseControlPlane(cp =>
{
    cp.WithHubUrl(builder.Configuration["Pragmatic:ControlPlane:HubUrl"]!);
    cp.WithApiKey(builder.Configuration["Pragmatic:ControlPlane:ApiKey"]!);
});

Or use appsettings.{Environment}.json:

appsettings.Development.json

{
  "Pragmatic": {
    "ControlPlane": {
      "HubUrl": "https://localhost:5100/_pragmatic/control-plane"
    }
  }
}

// appsettings.Production.json
{
  "Pragmatic": {
    "ControlPlane": {
      "HubUrl": "https://control-plane.internal:5100/_pragmatic/control-plane"
    }
  }
}

---

2. Missing API Key in Production

Symptom: Anyone on the network can connect to the control plane hub, send commands, or view host status.

Cause: No API key configured on the hub. The ControlPlaneAuthFilter only validates keys when one is configured.

// Wrong: no authentication
builder.UseControlPlaneHub();

// Correct: require API key
builder.UseControlPlaneHub(hub =>
{
    hub.WithApiKey(builder.Configuration["Pragmatic:ControlPlane:ApiKey"]!);
});

Rule: Always configure an API key in production. Store it in a secret manager (Azure Key Vault, AWS Secrets Manager, Kubernetes secrets), not in appsettings.json.

Both the hub and all clients must use the same API key:

// Hub
hub.WithApiKey("my-cluster-secret");

// Client
cp.WithApiKey("my-cluster-secret");

---

3. Self-Connection Deadlock in Embedded Hub

Symptom: The host hangs at startup. Logs show the SignalR client trying to connect but the hub is not listening yet.

Cause: The client tries to connect to the embedded hub before Kestrel has started listening.

Why this usually does NOT happen: ControlPlaneConnectionService registers a callback on IHostApplicationLifetime.ApplicationStarted, which fires only after Kestrel is ready. The connection is deferred automatically.

When it CAN happen: If you manually call SignalRControlPlane.ConnectAsync() in a startup step or IStartupStep.ConfigureServices(), before the host pipeline is built.

// Wrong: connecting too early
public class MyStartupStep : IStartupStep
{
    public void ConfigureServices(IServiceCollection services)
    {
        // DON'T resolve and connect here
        var cp = services.BuildServiceProvider().GetRequiredService<SignalRControlPlane>();
        cp.ConnectAsync().Wait(); // Deadlock!
    }
}

Fix: Let ControlPlaneConnectionService handle the connection lifecycle. It waits for ApplicationStarted automatically.

---

4. Mismatched Heartbeat and Stale Eviction Intervals

Symptom: Hosts are evicted from the registry even though they are running and healthy. Logs show: Host disconnected: {HostId} followed by re-registration.

Cause: The stale eviction interval is shorter than or too close to the heartbeat interval.

// Wrong: eviction fires before heartbeat arrives
hub.WithStaleEvictionInterval(TimeSpan.FromSeconds(15));
cp.WithHeartbeatInterval(TimeSpan.FromSeconds(15));  // Race condition!

// Correct: eviction interval = 4x heartbeat (minimum 2x)
hub.WithStaleEvictionInterval(TimeSpan.FromMinutes(2));
cp.WithHeartbeatInterval(TimeSpan.FromSeconds(30));

Rule of thumb: Set StaleEvictionInterval to at least 4x HeartbeatInterval. This accounts for network latency, GC pauses, and transient hub unavailability.

---

5. Forgetting to Register Health Contributors

Symptom: The hub shows all hosts as healthy, but one is actually experiencing database issues. Health is always “Healthy” with no detail.

Cause: No IHostHealthContributor implementations are registered. The default health status is “Healthy” when no contributors are present.

// Wrong: no health contributors
// The host reports "Healthy" even if the database is down.

// Correct: register contributors for critical dependencies
services.AddSingleton<IHostHealthContributor, DatabaseHealthContributor>();
services.AddSingleton<IHostHealthContributor, CacheHealthContributor>();
services.AddSingleton<IHostHealthContributor, MessageBusHealthContributor>();

Each contributor checks a specific dependency and returns ContributorHealthStatus.Healthy, Degraded, or Unhealthy. The aggregator combines them into the overall host health.

---

6. Not Handling Graceful Degradation in Application Code

Symptom: Application logic breaks when the control plane hub goes down. Errors like HubException: Connection closed propagate to request handlers.

Cause: Application code assumes IControlPlane is always connected and does not handle the disconnected case.

// Wrong: assumes connection
var hosts = await controlPlane.GetAllHostsAsync(ct);
var leaderHost = hosts.First(h => h.State == HostState.Serving);  // throws if empty

// Correct: handle degraded mode
var hosts = await controlPlane.GetAllHostsAsync(ct);
// In NoOp/degraded mode, this returns only the current host
if (hosts.Count == 0) return;  // or handle appropriately

SignalRControlPlane already handles this internally — if not connected, methods return sensible defaults (e.g., GetAllHostsAsync returns only self). But your application code should not assume the full cluster roster is always available.

---

7. Running Multiple Hub Instances Without a Backplane

Symptom: Hosts connected to hub instance A cannot see hosts connected to hub instance B. Commands sent to hosts on the other instance fail with “Host not connected”.

Cause: The in-memory ControlPlaneHubRegistry is per-process. Without a SignalR backplane, each hub instance has its own isolated registry.

Fix: Add a SignalR backplane for multi-hub deployments:

// Redis backplane
builder.Services.AddSignalR()
    .AddStackExchangeRedis("redis:6379", options =>
    {
        options.Configuration.ChannelPrefix = RedisChannel.Literal("pragmatic-cp");
    });

// Azure SignalR Service
builder.Services.AddSignalR()
    .AddAzureSignalR("Endpoint=...;AccessKey=...");

For most deployments: A single hub instance is sufficient. The backplane is only needed for high-availability hub deployments.

---

8. Confusing DatabaseLeaderElection with Hub-Based Leadership

Symptom: Migrations run twice — once via DatabaseLeaderElection and once via the hub’s ClaimMigrationLeadership.

Cause: Both leader election mechanisms are active, and they do not coordinate with each other.

How it actually works: DatabaseLeaderElection is used by MigrationRunner and operates at the database level (via __PragmaticLock). The hub’s ClaimMigrationLeadership is an additional coordination layer at the cluster level.

In practice, they are complementary:

1. Hub leadership (cluster level): Determines which host instance attempts migration
2. Database leadership (__PragmaticLock): Prevents concurrent migrations even if hub leadership fails

If you use both, the hub leadership prevents most contention, and the database lock acts as a safety net.

When to use which:

Scenario	Recommendation
Single instance, no hub	DatabaseLeaderElection (automatic)
Multiple instances, with hub	Hub leadership + DatabaseLeaderElection as safety net
Multiple instances, no hub	DatabaseLeaderElection only (automatic)

Rule: Do not disable DatabaseLeaderElection even when using the hub. It is the last line of defense against concurrent migrations.

---

Quick Reference

#	Mistake	Fix
1	Hardcoded hub URL	Use appsettings.{Environment}.json
2	No API key in production	Configure via secret manager
3	Early self-connection in embedded hub	Let ControlPlaneConnectionService handle timing
4	Eviction interval too close to heartbeat	Eviction = 4x heartbeat minimum
5	No health contributors	Register IHostHealthContributor for each dependency
6	Assuming always-connected	Handle degraded mode (NoOp returns)
7	Multiple hubs without backplane	Add Redis or Azure SignalR backplane
8	Confusing DB vs hub leadership	Use both; DB lock is the safety net