Table of Contents

Large-volume import and data adjustment workflows

Overview

This article explains how to implement and run large-volume import and data-adjustment workflows. It focuses on throughput, memory control, cancellation, and save pipeline configuration.

The example is based on:

  • A background server method for long-running processing.
  • Batch-oriented writes with a new service scope per batch.
  • Optional tuning of save execution through SaveOptions.

For large imports, use this execution flow:

  1. Trigger a start method from the UI.
  2. Start a background method to execute the import.
  3. Process records in batches.
  4. Recreate the scope per batch to limit memory pressure.
  5. Persist each batch with explicit save settings.
  6. Emit progress and completion notifications.
flowchart TD
    UI[UI action] --> START[Start server method]
    START --> BG[Background server method]
    BG --> LOOP{Remaining records?}
    LOOP -->|Yes| SCOPE[Create IServiceScope]
    SCOPE --> BUILD[Build batch entities]
    BUILD --> SAVE[SaveAsync with SaveOptions]
    SAVE --> NOTIFY[Send progress notification]
    NOTIFY --> LOOP
    LOOP -->|No| END[Send completion notification]

Batch and scope strategy

Use a short-lived scope for each batch to avoid a long-lived Entity Framework context. This helps keep memory usage stable during large imports.

while (created < orderCount)
{
    using IServiceScope scope = _serviceScopeFactory.CreateScope();

    IOrderRepository orderRepository = scope.ServiceProvider.GetRequiredService<IOrderRepository>();
    IUnitOfWork unitOfWork = scope.ServiceProvider.GetRequiredService<IUnitOfWork>();

    int batchCount = Math.Min(batchSize, orderCount - created);
    for (int index = 0; index < batchCount; index++)
    {
        Order order = CreateOrder(created + index);
        orderRepository.Add(order);
    }

    Result result = await unitOfWork.SaveAsync(cancellationToken);
    if (result.IsFailed)
    {
        throw new BusinessException(string.Join(Environment.NewLine, result.Errors.Select(error => error.Message)));
    }

    created += batchCount;
}

Save options for performance-sensitive workflows

For heavy technical imports, you can reduce overhead by configuring save options:

SaveOptions is available starting with Neos 3.1.

If validation rules execute SQL queries, disabling validation rules can significantly improve throughput. Use this only for controlled technical operations where compensating checks are planned.

SaveOptions options = new()
{
    EnableValidationRules = true,
    EnableEventRules = false,
    EnableQueryLogging = false,
};

Result result = await unitOfWork.SaveAsync(options, cancellationToken);

When you need full exception safety in long-running jobs, combine options with SaveMode.NeverThrow:

Result result = await unitOfWork.SaveAsync(
    SaveMode.NeverThrow,
    new SaveOptions
    {
        EnableValidationRules = true,
        EnableEventRules = false,
        EnableQueryLogging = false,
    },
    cancellationToken);
Warning

Disabling event rules or validation rules is high risk and can silently bypass critical business protections. This can produce invalid or inconsistent data that is expensive to detect and fix later. Use this only for tightly controlled technical operations, with explicit validation and rollback plans.

Post-migration data adjustment use case

These options can be useful after a database migration when you need to run corrective or enrichment treatments on existing data.

In this context, temporary rule deactivation can prevent technical treatments from being blocked by business rules designed for interactive business flows.

Caution

Post-migration bypasses must be temporary, traceable, and reviewed. Always re-enable rules after the treatment and run integrity checks before opening the system to users.

Cancellation and monitoring

For robust operations:

  • Check cancellation token regularly.
  • Distinguish cancellation from functional errors in notifications.
  • Track elapsed time and processed records for diagnostics.
  • Prefer informational notifications for user-requested cancellation.

Tuning approach

There is no universal numeric default for batch processing. The right values depend on your infrastructure, data model complexity, and business rules.

Use an iterative approach:

  1. Start with a controlled test scope.
  2. Monitor duration, memory usage, and error patterns.
  3. Change one parameter at a time (batch size, total volume, logging level, or rule settings).
  4. Keep the last stable configuration and document it for your environment.
Note

Avoid copying numeric values from another project without validation. Always validate settings against your own performance and reliability targets.

See also