Build a complete background processing workflow

This article describes a production-ready pattern to implement long-running background processing in Neos. It is based on the DashboardGeneration reference implementation available in the Technical Demos cluster.

Important

The concepts of batch and task used throughout this article are specific to this example. They represent one way to model multi-step background work, but they are not a framework requirement. Your own implementation may use a single entity, a different state vocabulary, or a different decomposition entirely. Focus on the design decisions, not the exact data model.

When to use this pattern

Use this pattern when you need to:

Execute long-running operations without blocking the UI.
Track progress across one or more concurrent operations.
Support retries and final failure handling.
Keep UI state synchronized in real time.

If your background work is a one-shot fire-and-forget with no state to track, a simple background server method is sufficient. This pattern targets workflows where observability and correctness under failure are required.

Key design decisions

When designing a background workflow, several non-obvious questions arise. This section addresses each one directly before presenting the implementation.

Should I persist state?

Yes, always.

Background methods execute in a separate auto-generated process (TaskRunner) that the main application cannot query directly. There is no API to interrogate a running TaskRunner instance. Persisting state is therefore the only reliable way to:

Show progress to users.
Recover from crashes or service restarts.
Audit what happened during a run and why it failed.
Detect that a new attempt should invalidate an older one.

What kind of data to persist (based on the reference example — adapt field names and states to your domain):

A lifecycle state to track where the operation stands (NotStarted, InProgress, Completed, Failed… the exact vocabulary is yours to define).
A correlation token (called BackgroundTaskIdentifier in the example): a freshly generated Guid string created for each execution attempt.
Timestamps for monitoring and diagnostics.
A diagnostic payload with error details in human-readable form, useful for support and retry decisions.

The correlation token is the most important field. Every state manager method filters on this token so a stale callback from a previous attempt cannot apply a transition to a task that has already moved on.

Important

Persist and commit the BackgroundTaskIdentifier before launching the background runner. If the runner starts before the identifier is committed, the first callbacks will arrive before the token is visible to other database connections and will be silently discarded.

How do I prevent concurrent conflicts?

In distributed systems, multiple workers can race to update the same record. The naive approach — load the entity, check state in memory, save — creates a race window and leads to hard-to-reproduce bugs.

Do not use load-modify-save for transitions:

// ❌ Unsafe: another worker may have changed state between Load and Save
var task = await _repo.GetByIdAsync(taskId);
if (task.State == NotStarted)
{
    task.State = InProgress;
    await _repo.SaveChangesAsync();
}

Use conditional direct updates instead:

// ✅ Safe: the condition is evaluated atomically inside a single SQL UPDATE
int updatedRowCount = await _repo.GetQuery()
    .Where(t => t.Id == taskId
        && t.BackgroundTaskIdentifier == backgroundTaskIdentifier
        && t.State == NotStarted)
    .ExecuteUpdateAsync(t => t
        .SetProperty(t => t.State, InProgress)
        .SetProperty(t => t.StartedAt, now));

ExecuteUpdateAsync translates directly to a single SQL UPDATE ... WHERE ... statement. The number of affected rows becomes your transition result:

1 — the transition was applied; proceed.
0 — the condition was not met (stale attempt, wrong state, concurrent write); stop quietly.

For aggregate transitions (marking a batch complete when all tasks are done), push the child-state check into the same WHERE clause to avoid TOCTOU (time-of-check/time-of-use) races:

// Atomically checks that no task is still pending before marking the batch complete
.Where(b => b.Id == batchId && !b.Tasks.Any(t => t.State != Completed))

If two tasks complete at the same time, only one of these statements will match — that worker finalizes the batch; the other gets updatedRowCount == 0 and stops.

How do I keep the UI up to date?

Because the TaskRunner is an auto-generated, isolated process with no queryable API, polling the background process directly is not possible. The only viable approach is server push: the background task writes state to the database and then notifies connected clients via SignalR.

This pattern uses server push via SignalR notifications.

After each successful state transition, emit a lightweight progress notification:

if (updatedRowCount == 1)
{
    await _progressNotification.SendToCurrentTenantConnectionsAsync(
        new DashboardGenerationProgressNotificationArguments
        {
            BatchId = batchId,
            Date = now,
            State = newState,
        }, cancellation);
}

The updatedRowCount == 1 guard is essential: never emit a notification for a transition that was not applied. Sending a notification when the update made no changes would push an incorrect state to connected UI clients.

On the frontend, UI event rules subscribe to this notification and refresh their data source. Because network delivery order is not guaranteed, apply an out-of-order guard based on the notification timestamp:

if (args.Date < lastDate)
{
    return; // Arrived out of order — discard
}
lastDate = args.Date;
// Refresh UI binding for this batch/task

Without this guard, a late-arriving InProgress notification could overwrite a Completed state that the user is already seeing.

How do I handle failures and retries?

Two distinct failure scenarios must be handled differently.

Transient failure (retryable)

A task fails but may succeed on another attempt. Handle this in the execution template:

Catch the exception.
Persist Failed state with diagnostic information via TrySetTaskFailedAsync.
Rethrow the original exception.

Rethrowing is critical. It signals the scheduler that this execution attempt failed, which triggers the configured retry policy. If you swallow the exception, the scheduler considers the task successful and will never retry.

catch (Exception ex)
{
    bool persisted = await _stateManager.TrySetTaskFailedAsync(
        batchId, taskId, backgroundTaskIdentifier, ex, cancellationToken);
    if (!persisted)
    {
        return; // Stale attempt — stop without rethrowing
    }
    throw; // Let the scheduler retry policy take over
}

Note that if TrySetTaskFailedAsync returns false, this attempt was already superseded by another one. In that case, do not rethrow — the new attempt owns this task and you must not interfere.

Final failure (non-retryable)

When all retry attempts are exhausted, the scheduler calls a designated error handler (HandleDashboardGenerationTaskError). This handler must:

Mark the task as FinalFailed (permanent, no more retries).
Evaluate whether all sibling tasks have reached a terminal state.
If so, mark the batch as FinalFailed too.

Important

When retrying a task, always assign a new BackgroundTaskIdentifier. This invalidates all in-flight callbacks from the previous attempt, preventing an older runner from writing over the freshly reset state.

The retry lifecycle in sequence:

Attempt 1 → exception → TrySetTaskFailedAsync → rethrow → scheduler retries
Attempt 2 (new identifier) → exception → TrySetTaskFailedAsync → rethrow → scheduler retries
Attempt 3 (new identifier) → exception → TrySetTaskFailedAsync → rethrow
  → retries exhausted → HandleErrorMethod → TrySetTaskFinalFailedAsync + TrySetBatchFailedAsync

What if I don't need retries?

If your use case does not require retries, the error handler method is unnecessary. Simplify by moving the final-failure logic directly into the catch block of the execution template:

catch (Exception ex)
{
    // Persist final failure state inline — no separate error handler needed
    await _stateManager.SetOperationFailedAsync(operationId, ex, cancellationToken);
    // Do not rethrow: no retry is expected, the operation is permanently failed
}

In this simplified form, the background method fully owns the failure path and no scheduler error hook needs to be registered.

Always provide a user-initiated recovery action

Automatic retry policies only fire when an exception propagates out of the background method. They cannot help when a task ends up stuck in an unexpected state for reasons outside the normal execution path:

The TaskRunner process was killed mid-execution, leaving the task in InProgress with no runner active.
A state store outage prevented the runner from starting at all, leaving the task in NotStarted indefinitely.
A bug caused the background method to return without updating state.

In all these cases, no exception is thrown, so no automatic recovery occurs. The task is permanently stuck unless the user can force a reset.

Important

Always expose a user-accessible action that can reset a stuck task and relaunch it. Without this escape hatch, an infrastructure failure becomes a permanent data issue that requires manual database intervention.

The recovery action must:

Assign a new correlation token to the task to invalidate any lingering in-flight callbacks.
Reset the task state to the initial state (NotStarted in the reference example).
Persist the changes.
Relaunch the background runner.

This is exactly what RetryDashboardGenerationTask does in the reference implementation — and it covers both the normal case (user retrying a Failed task) and the abnormal case (user unsticking a task that never progressed).

Note that this action is intentionally permissive: it does not restrict which state a task must be in before allowing a reset. restricting it (for example, to Failed only) would leave users helpless when a task is stuck in InProgress.

Must my task be idempotent?

Yes. This requirement applies to all background methods, not just complex workflows. See Idempotency in the background server methods article for the full explanation and techniques.

In the context of this pattern, note that the correlation token guards state transitions only. The business logic inside the task (the equivalent of GenerateMarkdownAsync) is specific to your domain and must be made idempotent explicitly, independently of the state machine.

How do I structure my code for maintainability?

Centralize all transitions in a single state manager.

A dedicated component (IDashboardGenerationStateManager) owns every state transition. This has several benefits:

Single place to audit, test, and evolve state logic.
Consistent concurrency and notification handling across all transitions.
Prevents accidental direct writes from spreading across the codebase.

Warning

ExecuteUpdateAsync is an extension method on IQueryable<T> that issues a direct SQL statement. It cannot be mocked: calling it in a unit test that uses a repository mock will throw an exception. Methods that call ExecuteUpdateAsync are therefore not unit-testable.

To limit the amount of untestable code, isolate all ExecuteUpdateAsync calls in a dedicated service (the state manager in this example) and keep these methods as thin as possible — ideally just the conditional update and the notification call. Everything else (business decisions, orchestration logic) belongs in code that can be tested normally.

Use a base class to define the execution template.

All task-specific background methods share the same lifecycle: acquire run right, produce payload, persist result, handle failure. Encoding this sequence in a base class (DashboardGenerationTaskBase) ensures that all tasks follow consistent behavior without duplicating orchestration code. Concrete tasks only implement GenerateMarkdownAsync — the business-specific part.

Wrap orchestration methods in transactions.

Commit the transaction before launching runners. This ensures that background workers always read committed identifiers and state from the database. If runners were launched inside an open transaction, they could attempt to read task records before the identifiers are visible to other database connections, causing the first callbacks to be silently discarded.

Start a database transaction.
Apply state changes and save.
Commit.
Launch runners after commit.

If a runner launch fails after the commit, the state changes are already durably stored: identifiers and task states remain in the database. The affected tasks stay in NotStarted and can be individually rescheduled via the Retry command without any data loss.

Register task definitions with dual DI scope.

Each task definition must be registered twice:

// Unkeyed: for IEnumerable<IDashboardGenerationTaskDefinition> injection at batch creation time
services.AddScoped<IDashboardGenerationTaskDefinition, OperationsDashboardTaskDefinition>();

// Keyed: for runtime resolution by task.Code when a specific task must be restarted
services.AddKeyedScoped<IDashboardGenerationTaskDefinition, OperationsDashboardTaskDefinition>(
    typeof(OperationsDashboardTaskDefinition).Name);

The keyed registration uses the type name as the key, which is also the value persisted in task.Code. This allows a restarted or retried task to be resolved from only its persisted string code, with no hardcoded switch/case.

Reference implementation

The following files form the full reference example:

File	Role
`Methods/CreateDashboardGenerationBatch.cs`	Creates batch and task rows; no background work started here
`Methods/StartDashboardGenerationBatch.cs`	Assigns identifiers, persists, launches runners
`Methods/RetryDashboardGenerationTask.cs`	Assigns new identifier, resets state, relaunches
`Methods/HandleDashboardGenerationTaskError.cs`	Final failure handler hooked by the scheduler
`Tasks/Infrastructure/DashboardGenerationTaskBase.cs`	Execution template shared by all tasks
`Tasks/Infrastructure/DashboardGenerationStateManager.cs`	All state transitions and UI notifications
`Tasks/Infrastructure/IDashboardGenerationTaskDefinition.cs`	Task definition contract
`Tasks/Infrastructure/ServiceCollectionExtensions.cs`	Dual-scope DI registration

All paths are relative to demos/technicaldemos/modules/DashboardGeneration/businessAssembly/Application/.

UI synchronization is implemented in the UIProject:

UIs/DashboardGenerationBatchUI/DashboardGenerationBatchUIEventRules.cs
UIs/DashboardGenerationTaskUI/DashboardGenerationTaskUIEventRules.cs

Target architecture

The pattern separates responsibilities into four layers:

Orchestration methods: create / start / retry / final-error flows.
Task definitions: map persisted task code to background runners.
Execution template: shared state-machine behavior via a base class.
State manager: atomic transitions and UI notifications.

flowchart TD
    A[User action] --> B[Create batch & tasks]
    B --> C[Start batch]
    C --> D[Assign new BackgroundTaskIdentifier per task]
    D --> Commit[Commit transaction]
    Commit --> E[Launch background runners]
    E --> F[TaskBase: TryStartTaskAsync]
    F --> G[GenerateMarkdownAsync]
    G --> H[TryCompleteTaskAsync]
    H --> I[TryCompleteBatchAsync]

    G --> J[Exception]
    J --> K[TrySetTaskFailedAsync]
    K --> L{Retries left?}
    L -->|Yes| D
    L -->|No| M[HandleErrorMethod]
    M --> N[TrySetTaskFinalFailedAsync + TrySetBatchFailedAsync]

    H --> O[Progress notification]
    K --> O
    N --> O
    O --> P[UI event rules refresh data]

Common pitfalls

Pitfall	Impact	Correct approach
Using load-modify-save in concurrent paths	Race conditions, double-apply	Conditional direct updates with `ExecuteUpdateAsync`
Reusing the same identifier on retry	Stale callbacks corrupt new attempt	New `BackgroundTaskIdentifier` per execution attempt
Emitting notifications for updates that made no changes	UI receives incorrect state	Check `updatedRowCount == 1` before notifying
Checking child state before batch update	TOCTOU race at completion	Embed child condition inside the batch `WHERE` clause
Swallowing exceptions after persisting failed state	Scheduler never retries	Always rethrow after `TrySetTaskFailedAsync`
Launching runners before committing the transaction	Worker reads stale snapshot; callbacks discarded	Commit first, then launch runners outside the transaction
Launching runners before persisting identifier	First callbacks arrive before token exists	Persist and commit first, then launch
Skipping out-of-order guard in UI event rules	Late arrival rolls back displayed state	Compare `args.Date` with last received date
Writing non-idempotent business logic	Duplicates and corrupted data on any re-execution	Design every task to be safely re-runnable with the same inputs

Minimal checklist

[ ] Model your entities with a lifecycle state, a correlation token per execution attempt, and timestamps.
[ ] Persist the BackgroundTaskIdentifier before calling any runner.
[ ] Register task definitions unkeyed (enumeration) and keyed by type name (runtime resolution).
[ ] Use ExecuteUpdateAsync for all state transitions; branch on updatedRowCount.
[ ] Emit progress notifications only when updatedRowCount == 1.
[ ] Apply timestamp out-of-order guard in UI event rules.
[ ] Rethrow exceptions after persisting failed state.
[ ] Assign a new identifier on every retry.
[ ] Call TrySetTaskFinalFailedAsync and batch finalization in the error handler.
[ ] Wrap state changes in a transaction; commit before launching runners.
[ ] Design business logic inside each task to be idempotent: safe to re-run with the same inputs.

Table of Contents

Build a complete background processing workflow

Important

When to use this pattern

Key design decisions

Should I persist state?

Important

How do I prevent concurrent conflicts?

How do I keep the UI up to date?

How do I handle failures and retries?

Transient failure (retryable)

Final failure (non-retryable)

Important

What if I don't need retries?

Always provide a user-initiated recovery action

Important

Must my task be idempotent?

How do I structure my code for maintainability?

Warning

Reference implementation

Target architecture

Common pitfalls

Minimal checklist

See also