Troubleshooting

Practical problem/solution guide for Pragmatic.Jobs. Each section covers a common issue, the likely causes, and the fix.

---

Job Not Executing

You have a job class that compiles, but it never runs at runtime.

Checklist

1. Does the class have [Job] or [RecurringJob]? Without the attribute, the SG does not discover the class. It compiles fine but has no invoker, no DI registration, and no type registry entry.

2. Is the class partial? Diagnostic PRAG2502 warns if missing. Without partial, the invoker is generated in a degraded mode.

3. Does the class implement IJob or IJob<T>? Diagnostic PRAG2500 fires if the interface is missing. The SG will not generate an invoker.

4. Is the SG analyzer referenced? In your .csproj, the Pragmatic.SourceGenerator must be referenced with OutputItemType="Analyzer":

   <ProjectReference Include="..\Pragmatic.SourceGenerator\...\Pragmatic.SourceGenerator.csproj"
                      OutputItemType="Analyzer"
                      ReferenceOutputAssembly="false" />

5. Are processing services registered?

   With Composition (automatic):

   await PragmaticApp.RunAsync(args, app =>
   {
       app.UseJobs();
   });

   Without Composition (manual — both calls are required):

   builder.Services.AddPragmaticJobs();
   builder.Services.AddJobProcessingServices();  // Registers the BackgroundServices

6. Check the SG output. In Visual Studio, expand Dependencies > Analyzers > Pragmatic.SourceGenerator in Solution Explorer. Look for {JobType}.Invoker.g.cs and _Infra.Jobs.Registration.g.cs. If these files do not exist, the SG is not processing your class.

7. Is NullJobTypeRegistry in use? If no jobs are discovered, the runtime IJobTypeRegistry resolves to NullJobTypeRegistry, and both JobProcessorService and RecurringJobSchedulerService exit immediately. Check that the SG-generated PragmaticJobTypeRegistry is registered.

8. Is the ScheduledFor time in the future? JobProcessorService only fetches jobs where ScheduledFor <= now. If you scheduled a delayed job, verify the timestamp.

9. Check the logs. JobProcessorService logs at startup: "Job processor started: worker={WorkerId}, concurrency={WorkerCount}". If you do not see this message, the service is not running. Look for "Job processor poll error" messages that indicate exceptions during polling.

---

Recurring Job Not Registered

A recurring job exists in code but does not appear in __RecurringJobs and never fires.

Checklist

1. Does the class have [RecurringJob("cron")]? Not [Job] — only [RecurringJob] generates entries in _Infra.Jobs.RecurringJobs.g.cs.

2. Is the cron expression valid? An empty or malformed expression triggers PRAG2501. Check the build output for this diagnostic.

3. Check _Infra.Jobs.RecurringJobs.g.cs. Open the SG output and verify your job appears in the RegisterRecurringJobs method.

4. Is the recurring definition upserted at startup? The SG-generated registration calls IRecurringJobStore.UpsertAsync() for each recurring definition. If the store is EF Core, check the __RecurringJobs table.

5. Is RecurringJobSchedulerService running? Check the logs for "Recurring job scheduler started". If absent, AddJobProcessingServices() was not called.

6. Is the timezone correct? If you specify TimeZone = "America/New_York", the cron expression is evaluated in that timezone. A job scheduled for "0 2 * * *" in New York might not fire at the UTC time you expect.

7. Is the definition disabled? RecurringJobDefinition.IsEnabled defaults to true, but if something calls IRecurringJobStore.DisableAsync(), the job will not be picked up.

---

Lease Conflicts (Multiple Workers)

The pragmatic.jobs.lease_conflicts counter is high, meaning workers are competing for the same jobs.

Checklist

1. Is the batch size too large? If BatchSize = 100 and WorkerCount = 2, each poll fetches far more jobs than the workers can process before the next poll. Reduce BatchSize to WorkerCount * 2 or WorkerCount * 3.

2. Is the polling interval too short? If multiple instances poll every second, lease conflicts increase. The default 5 seconds is usually sufficient.

3. Are leases expiring too fast? If LeaseTimeSeconds is shorter than the typical job execution time, the lease expires mid-execution. Another worker picks up the same job. Increase LeaseTimeSeconds to be at least 2x the expected maximum job duration.

---

Diagnostics Reference

The source generator emits the following diagnostics for common mistakes:

ID	Severity	Title	Message
PRAG2500	Error	Job class must implement IJob or IJob	Type ‘{0}’ is decorated with [Job]/[RecurringJob] but does not implement IJob or IJob
PRAG2501	Error	Invalid cron expression	[RecurringJob] on ‘{0}’ has an empty or missing cron expression
PRAG2502	Warning	Job class should be partial	Type ‘{0}’ is decorated with [Job]/[RecurringJob] but is not declared as partial
PRAG2503	Error	Duplicate recurring job ID	Recurring job ID ‘{0}’ is used by both ‘{1}’ and ‘{2}‘
PRAG2504	Error	Invalid retry configuration	[Retry] on ‘{0}’ has MaxAttempts = {1}. MaxAttempts must be greater than 0
PRAG2505	Error	Continuation must be a job	[Continuation<{0}>] on ‘{1}’: type ‘{0}’ does not implement IJob or IJob
PRAG2506	Error	Continuation cycle detected	Job ‘{0}’ creates a cycle in continuation chain: {1}

Reading the Diagnostics

• PRAG2500: Add : IJob or : IJob<TParams> to your class declaration.
• PRAG2501: Provide a valid 5-part cron expression. Empty strings and expressions with the wrong number of fields are rejected.
• PRAG2502: Add the partial keyword to your class declaration.
• PRAG2503: Specify a unique Id in [RecurringJob(cron, Id = "unique-id")], or rely on auto-generated IDs by omitting the Id property.
• PRAG2504: Set MaxAttempts to a positive integer. If you want no retry, omit [Retry] entirely.
• PRAG2505: The type argument in [Continuation<T>] must implement IJob or IJob<T>. Check the target class declaration.
• PRAG2506: Remove [Continuation<T>] from one of the jobs in the cycle to break it. The diagnostic message shows the full chain path.

---

FAQ

Q: Can I use [RecurringJob] and [Job] on the same class?

No. A class should have either [RecurringJob] (for cron-based scheduling) or [Job] (for on-demand scheduling), not both. If you need the same job logic to run on a schedule and also on demand, extract the logic into a shared service and create two separate job classes.

Q: What happens if my job throws an exception?

If [Retry] is configured, the generated invoker catches the exception and retries with the configured backoff strategy. After all attempts are exhausted, JobProcessorService marks the job as Failed in the store. The error message is stored in JobInstance.Error.

If [Retry] is not configured, the default MaxAttempts from JobsOptions.DefaultMaxRetries applies (default: 1, meaning no retry). The job is marked Failed on the first exception.

Q: How do I cancel a scheduled job?

Use IJobScheduler.CancelAsync(jobId). The jobId is the Guid returned by ScheduleAsync or ScheduleAtAsync. This marks the job as Cancelled in the store. If the job is already running, cancellation depends on whether your ExecuteAsync observes the CancellationToken.

var jobId = await scheduler.ScheduleAsync<SendReminderJob, ReminderParams>(
    new ReminderParams(reservationId, email),
    delay: TimeSpan.FromHours(24));

// Later, if the reservation is cancelled:
await scheduler.CancelAsync(jobId);

Q: Can I have multiple workers processing different job types?

The current architecture uses a single JobProcessorService that processes all job types. Workers are generic — any worker can process any job. This is by design: the lease mechanism ensures each job is processed by exactly one worker. If you need dedicated workers for specific job types, deploy separate app instances with different job assemblies.

Q: How does the InMemory store behave with multiple workers?

InMemoryJobStore and InMemoryRecurringJobStore are singletons within a single process. They work with WorkerCount > 1 (multiple tasks in the same process) but do NOT work across multiple processes or containers. For multi-instance deployments, use UseEfCore().

Q: What happens if the cron timezone is invalid?

RecurringJobSchedulerService calls TimeZoneInfo.FindSystemTimeZoneById(), which throws TimeZoneNotFoundException for invalid IANA names on systems that do not support them. On Linux, IANA names are natively supported. On Windows, .NET 10 supports IANA names via ICU. If you see timezone errors, ensure your runtime supports the specified timezone.

Q: How do I see what jobs are pending/running/failed?

Query IJobStore directly. In EF Core mode, query the __Jobs table:

-- Pending jobs
SELECT * FROM __Jobs WHERE Status = 0 ORDER BY ScheduledFor;

-- Failed jobs with error details
SELECT Id, JobType, Error, Attempt, MaxAttempts FROM __Jobs WHERE Status = 3;

-- Currently running (leased) jobs
SELECT Id, JobType, LeasedBy, LeaseExpiresAt FROM __Jobs WHERE Status = 1;

Q: My continuation job does not receive the parameters from the parent job. Why?

Continuation jobs receive the CorrelationId from the parent job, not the parameters. If the continuation needs data from the parent job, either:

• Use CorrelationId to look up the data in a shared store.
• Store the result of the parent job in a database and query it in the continuation.

---

Observability Debugging

OpenTelemetry Spans Not Appearing

1. Is the ActivitySource configured? Add Pragmatic.Jobs to your OpenTelemetry tracing configuration:

   builder.Services.AddOpenTelemetry()
       .WithTracing(tracing => tracing
           .AddSource("Pragmatic.Jobs"));

2. Are spans being sampled out? Check your sampler configuration. With AlwaysOnSampler, all spans are captured.

Metrics Not Appearing

1. Is the Meter configured? Add Pragmatic.Jobs to your OpenTelemetry metrics configuration:

   builder.Services.AddOpenTelemetry()
       .WithMetrics(metrics => metrics
           .AddMeter("Pragmatic.Jobs"));

2. Metrics available: pragmatic.jobs.enqueued, pragmatic.jobs.completed, pragmatic.jobs.failed, pragmatic.jobs.retried, pragmatic.jobs.duration, pragmatic.jobs.lease_acquisitions, pragmatic.jobs.lease_conflicts, pragmatic.jobs.recurring_triggered.

---

Job Stuck in Running State

A job appears as Running (Status = 1) in __Jobs but no worker is actively processing it.

Checklist

1. Did the worker crash mid-execution? When a worker crashes, the lease remains until LeaseExpiresAt. JobProcessorService calls IJobStore.ReleaseExpiredLeasesAsync() at the start of each poll cycle. Once the lease expires, the job becomes eligible for reprocessing.

2. Is LeaseTimeSeconds too long? The default is 300 (5 minutes). If the worker crashed, you must wait for the full lease duration before another worker picks up the job. For fast jobs, reduce the lease time. For long jobs, the lease should exceed the expected execution time.

3. Check LeasedBy in the database. If the value matches a worker that no longer exists, wait for LeaseExpiresAt to pass. Alternatively, manually reset the job:

   UPDATE __Jobs
   SET Status = 0, LeasedBy = NULL, LeaseExpiresAt = NULL
   WHERE Id = 'your-job-id' AND Status = 1;

4. Is the job genuinely long-running? Check the StartedAt timestamp. If the job started recently and your [Timeout] is high, it may still be executing normally.

---

Job Parameters Deserialization Failure

A parametric job (IJob<T>) fails with a JSON deserialization error.

Checklist

1. Is the parameter type JSON-serializable? The SG-generated IJobTypeRegistry uses System.Text.Json for serialization. Ensure your parameter record/class has a parameterless constructor or uses record types (which serialize correctly by default).

2. Did the parameter type change after jobs were already enqueued? If you rename properties or change the type structure, already-enqueued jobs in __Jobs have the old JSON format. They will fail on deserialization. Clear or migrate old job records.

3. Are there unsupported types in the parameters? Avoid interface properties, Func<>, Expression<>, or circular references. Stick to simple records with primitive types, Guid, DateTimeOffset, string, and decimal.

---

EF Core Migration Issues

Tables Not Created

The __Jobs and __RecurringJobs tables are created by EfCoreJobStore and EfCoreRecurringJobStore entity configurations. If they do not appear:

1. Is UseEfCore() called? Check your UseJobs configuration:

   app.UseJobs(jobs => jobs.UseEfCore());

2. Was EnsureCreated or a migration run? The entity configurations are registered when the EF Core store is resolved. Ensure your database migration includes these tables.

3. Check the connection string. The EF Core stores use the same DbContext as your application. Verify the connection string points to the correct database.

---

Getting Help

• Showcase examples: See NoShowDetectionJob and SendCheckInReminderJob in examples/showcase/src/Showcase.Booking/Infrastructure/Jobs/ for working examples of both recurring and delayed jobs.
• Integration tests: examples/showcase/tests/Showcase.IntegrationTests/Jobs/ contains tests for job infrastructure and EF Core stores.
• Architecture: Read concepts.md for the full pipeline explanation and decision tree.
• Common mistakes: Read common-mistakes.md for the most frequent issues with code examples.