Table of Contents

Data import pipeline and runtime

This article documents the NeosDataExchange import architecture and runtime behavior for production-grade CSV imports.

It complements the overview in Data exchange with implementation-level concepts, processing stages, and extension points.

Scope

The NeosDataExchange import feature provides:

  • template-driven mapping from CSV columns to target entity-view properties,
  • staging-based processing with resumable background execution,
  • multi-stage validation and integration workflows,
  • support for master/detail imports through row grouping,
  • configurable save behavior for performance-sensitive imports.

End-to-end process

flowchart TD
  A[Import template<br>configuration] --> B[Upload CSV file]
  B --> C[Load staging rows]
  C --> D[Structure validation]
  D --> E[Business validation<br>or direct integration]
  E --> F[Integrate into<br>target entity view]
  F --> G[Update status, stage,<br>and summary]

Runtime architecture

The import runtime is built around three processing roles:

  • loader: reads source data and populates staging rows,
  • aggregator: groups and transforms staging payloads for integration,
  • integrator: applies business persistence rules to target entity views.
Note

These roles are registered by entity view through keyed DI registrations using RegisterImportEntityView.

flowchart LR
  UI[Import UI views] --> SM[Server methods]
  SM --> SI[Staging initializer]
  SM --> SL[Staging loader]
  SL --> AG[Staging aggregator]
  AG --> SV[Structure validator]
  SV --> IN[Staging integrator]
  IN --> EVI[Entity view integrator]
  EVI --> UOW[Unit of work save]
  UOW --> ST[(Staging tables)]
  UOW --> TT[(Target tables)]
  IN --> NT[Background notifications]
  NT --> UI

Pipeline stages and statuses

Imports progress through explicit pipeline stages (for example staging loading, grouping, structure validation, business validation, integration) and update row-level statuses (for example valid, information, warning, error, integrated).

This dual tracking enables:

  • accurate progress indicators for long-running imports,
  • retry/reset workflows for failed executions,
  • KPI-based summaries in staging detail screens,
  • selective continuation based on eligibility and execution mode.

UI workflow

The import UI is organized around dedicated views:

  • NeosDataImportTemplateListUI: template list and import history entry point,
  • NeosDataImportTemplateEditUI: template and mapping configuration,
  • NeosDataImportSettingsUI and NeosDataImportUploadFileUI: file and execution setup,
  • NeosDataImportStagingListUI: execution history and statuses,
  • NeosDataImportStagingEditUI: KPI summary, row details, and validation/integration actions.

The staging details grid supports dynamic columns from raw imported data to help diagnose mapping and validation issues.

Validation and integration behavior

Validation and integration behavior is controlled by template and execution options:

  • validation mode (simulation or run),
  • update kind,
  • batch size,
  • save options (EnableValidationRules, EnableEventRules, EnableQueryLogging).

This allows teams to run safe validation passes before business integration, and to tune performance for large imports.

Master/detail imports

Master/detail imports are handled through row grouping:

  • staging rows are grouped by a computed key,
  • grouped payloads are aggregated into import-ready mapped data,
  • integration runs at group level while preserving row-level traceability.

This model supports scenarios where a single source file contains repeated header values and multiple detail lines.

File and mapping capabilities

Current import capabilities include:

  • CSV upload through file selection and drag-and-drop,
  • header extraction to assist mapping,
  • target property discovery per import type,
  • multiple mapping enhancements (including ordering and additional mapping metadata),
  • string length validation based on target property constraints.

Operational considerations

When operating high-volume imports:

  • prefer explicit batch sizing aligned with database throughput,
  • use simulation mode to validate templates before run mode,
  • enable query logging only when troubleshooting,
  • monitor stage/status transitions to identify bottlenecks.

See also