Table of Contents

Customize the local collector, Jaeger and Prometheus

Neos installs a local Jaeger instance, a local OpenTelemetry Collector, and a local Prometheus instance in the user profile when you run neos setup. When the local observability tooling is started in the development environment, all executables are launched with configuration files stored in the .neos folder of the current user.

This article explains which files you can customize, what they are used for, and the main precautions to keep in mind.

Configuration files

The local OpenTelemetry Collector configuration file is stored at:

%USERPROFILE%\.neos\otelcol\otel-collector-config.yaml

This file is generated when Neos installs or updates the local tracing tools. It is rewritten only when the otelcol-contrib executable is missing, invalid, or on a version different from the one expected by the application. If the installed version is already valid, the existing configuration file is kept as is.

Use this file to customize how telemetry is collected, transformed, and exported locally. Typical changes include:

  • Adding or changing receivers
  • Adding processors
  • Changing exporters
  • Updating service pipelines
  • Forwarding telemetry to another backend such as Jaeger or Azure Monitor / Application Insights

The local Jaeger configuration file is stored at:

%USERPROFILE%\.neos\jaeger\config.yaml

The same principle applies to this file. It is generated when Neos installs or updates the local Jaeger tool, and it is rewritten only when the jaeger executable is missing, invalid, or on a version different from the one expected by the application. If the installed version is already valid, the existing configuration file is preserved.

Use this file to customize Jaeger-specific behavior, for example:

  • Storage configuration
  • Query service settings
  • OTLP receiver settings
  • Exposed ports and endpoints

The local Prometheus configuration file is stored at:

%USERPROFILE%\.neos\prometheus\config.yaml

The same principle applies to this file. It is generated when Neos installs or updates the local Prometheus tool, and it is rewritten only when the prometheus executable is missing, invalid, or on a version different from the one expected by the application. If the installed version is already valid, the existing configuration file is preserved.

Use this file to customize Prometheus-specific behavior, for example:

  • Scrape jobs and targets
  • Scrape intervals and timeouts
  • Relabeling rules
  • Remote write configuration

How Neos uses these files

When the local Jaeger tooling is started, Neos launches:

  • jaeger with the Jaeger configuration file
  • otelcol-contrib with the OpenTelemetry Collector configuration file

This means your local customizations are applied the next time Jaeger and the collector are restarted.

When the local Prometheus tooling is started, Neos launches prometheus with the Prometheus configuration file.

This means your local customizations are applied the next time Prometheus is restarted.

  1. Run neos setup if the local tooling is not installed yet.
  2. Edit the configuration files you want to customize.
  3. Restart local observability tooling in the local environment.
  4. Verify the result in Jaeger, Prometheus, or in the telemetry backend you configured.
Tip

The collector configuration file is the main entry point when you want to change how local traces are routed. For example, you can add processors, change exporters, or define additional telemetry pipelines. The Prometheus configuration file is the main entry point when you want to change local metrics scraping.

Example: customize local Prometheus scrape targets

You can adapt the local Prometheus configuration to scrape additional metrics endpoints during local investigation.

scrape_configs:
  - job_name: 'neos-otel'
    scrape_interval: 15s
    static_configs:
      - targets: ['127.0.0.1:8889']

See the official Prometheus documentation for more details on the configuration syntax and available options: Prometheus configuration.

Note

Adapt job names, targets, and intervals to your local setup. If your Prometheus web UI endpoint is unchanged, it is available at http://localhost:9080.

Example: add custom investigation attributes to all spans

When several developers share the same tracing backend, or when you need to isolate a specific debugging session, it can be useful to tag all exported spans with custom attributes.

receivers:
  zipkin:
    endpoint: "0.0.0.0:9412"

processors:
  attributes/investigation:
    actions:
      - key: neos.investigation
        value: local-debug-session
        action: upsert
      - key: neos.workstation
        value: DEVBOX01
        action: upsert
  batch:

exporters:
  otlp:
    endpoint: "0.0.0.0:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [zipkin]
      processors: [attributes/investigation, batch]
      exporters: [otlp]

This example keeps the Zipkin receiver used by local Neos tracing and adds:

  • An attributes processor that enriches all spans
  • A batch processor
  • The default otlp exporter to continue sending traces to Jaeger
Note

You can adapt the values of neos.investigation and neos.workstation to your own troubleshooting context. This makes the traces easier to identify in Jaeger or in another backend.

Example: filter out neos-state-store spans

When you investigate a trace with a large volume of inter-cluster communication, spans produced by neos-state-store can add noise and make the analysis harder to follow.

You can exclude these spans by adding a filter processor to the collector configuration:

processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - 'attributes["peer.service"] == "neos-state-store"'
  batch:

To apply the filter, also add it to the trace pipeline:

service:
  pipelines:
    traces:
      receivers: [zipkin]
      processors: [filter, batch]
      exporters: [otlp]

With this configuration, spans whose peer.service attribute is equal to neos-state-store are dropped before export.

Example: customize the local Jaeger instance

The generated Jaeger configuration uses in-memory storage. A common customization is to increase the number of traces kept in memory during local investigation.

Example:

service:
  telemetry:
    metrics:
      level: none
  extensions: [healthcheckv2, jaeger_storage, jaeger_query]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
    send_batch_max_size: 1024

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage

extensions:
  healthcheckv2:
    http:
      endpoint: 0.0.0.0:13133

  jaeger_storage:
    backends:
      some_storage:
        memory:
          max_traces: 50000

  jaeger_query:
    storage:
      traces: some_storage
    http:
      endpoint: 0.0.0.0:16686

Compared to the default generated configuration, the important change here is:

  • extensions.jaeger_storage.backends.some_storage.memory.max_traces: 50000

This can be useful when you need to keep more local traces available in Jaeger during a debugging session.

Important notes

Important

These files are stored in the user profile, not in the repository. They affect the local machine only.

Warning

neos setup does not rewrite these configuration files every time it runs. Neos rewrites a configuration file only when the corresponding tool must be installed or updated because its executable is missing, invalid, or on an unexpected version. Keep a copy of your custom configuration if you want to reapply it after such an installation or update.

Caution

Invalid YAML or unsupported settings can prevent Jaeger, the collector, or Prometheus from starting correctly.

Note

The examples in this article are starting points. Adjust receivers, processors, exporters, and endpoints to match your local observability setup.

Customize UI listening ports

By default, the local Jaeger UI is available at http://localhost:16686 and the local Prometheus UI is available at http://localhost:9080. To change these ports, you must follow these steps:

Jaeger

  • Edit the configuration file %USERPROFILE%\.neos\jaeger\config.yaml, search for
  jaeger_query:
    http:
      endpoint: 0.0.0.0:16686

and change the port to the desired value. For example, to listen on port 16687, change it to:

  jaeger_query:
    http:
      endpoint: 0.0.0.0:16687
  • Then add an environment variable Neos__Jaeger__ListenAddress with the desired url, e.g., 0.0.0.0:16687, so the url to open the Jaeger UI is consistent with the new port. This variable is also used by the neos setup command to not lose the configuration when it rewrites the Jaeger configuration file during an update.

Prometheus

  • Add an environment variable Neos__Prometheus__ListenAddress with the desired url, e.g., localhost:9090 to listen on port 9090.
Note

By default, Neos uses port 9080 for Prometheus, which is not a standard port. The main reason is that the standard port 9090 can be used by Darp placement process.

Important notes

Important

These files are stored in the user profile, not in the repository. They affect the local machine only.

Warning

neos setup does not rewrite these configuration files every time it runs. Neos rewrites a configuration file only when the corresponding tool must be installed or updated because its executable is missing, invalid, or on an unexpected version. Keep a copy of your custom configuration if you want to reapply it after such an installation or update.

Caution

Invalid YAML or unsupported settings can prevent Jaeger or the collector from starting correctly.

Note

The examples in this article are starting points. Adjust receivers, processors, exporters, and endpoints to match your local observability setup.

Configuration reference

For the collector configuration syntax and available options, see the official OpenTelemetry Collector documentation:

For Jaeger-specific settings, use the generated file as a starting point and refer to the official Jaeger documentation when needed:

For Prometheus-specific settings, use the generated file as a starting point and refer to the official Prometheus documentation when needed:

See also