Evaluation of AI Agents (Preview)

To evaluate the performance of an AI agent, we used the Microsoft.Extensions.AI.Evaluation library, you can see evaluation-libraries for more information.

Warning

Microsoft.Extensions.AI.Evaluation library is in preview. The API may change in the future.

Getting Started

To begin, you need to configure an LLM model to use for the evaluation. You can use the following code to configure the model. Currently, only OpenAI models are supported.

In the cluster YAML configuration file, you can add the following configuration to the "ai" section:

For example, to use the "gpt-5.4-mini" model for the "AzureOpenAI" provider, you can add the following configuration:

AI:
  Evaluation:
    Provider: AzureOpenAI
    ModelId: gpt-5.4-mini

The supported providers are :

AzureAIInference
AzureOpenAI
OpenAI

Then you must configure the secrets for the provider you are using. For example, for the AzureOpenAI provider, you can add the following configuration to the secrets section:

dotnet user-secrets set "AI:Evaluation:ApiEndpoint" "[Your_endpoint]" --id [Your_root_Namespace].AspNetCore
dotnet user-secrets set "AI:Evaluation:ApiKey" "[Your_Key]" --id [Your_root_Namespace].AspNetCore

The endpoint is the URL of your Azure AI deployment, and the key is the API key for that deployment.
You can find your cluster's root namespace in the YAML file at the root of your cluster.

RootNamespace: [Your_root_Namespace]

Evaluators

The library provides several evaluators to evaluate the performance of the AI agent. The following evaluators are available:

GroundednessEvaluator: This evaluator checks if the answer is relevant and true. It uses the evalution context to check if the answer is correct.
RelevanceTruthAndCompletenessEvaluator: Evaluates the extent to which a given response contains all necessary and relevant information with respect to the provided ground truth. It doesn't use the evalution context to check if the answer is correct.
CoherenceEvaluator: Measures the grammatical proficiency of a generative AI's predicted answer. It doesn't use the evalution context to check if the answer is correct.
FluencyEvaluator: Evaluates the fluency of a given response or a multi-turn conversation, including reasoning. It doesn't use the evalution context to check if the answer is correct.

Warning

The Microsoft.Extensions.AI.Evaluation library is in preview, so these evaluators may not be available in future versions of Neos.

Create the xUnit test project

In Neos Studio, in the AI Agent view, you can create an xUnit test project by clicking the "Create Evaluation Project" button in the toolbar. If the project doesn't exist, it will be created; otherwise, it will be updated. This action will create an xUnit class for the current agent. Each time you want to evaluate an agent, you must add a class using this action.

You can then write your tests in the generated class.

Note

A partial "Startup" class is created in the test project. This class is used to configure the test project's services. You can add your own services or replace some services with a mock. The "Startup.generated.cs" file containing the generated class part is generated by Neos Studio and should not be modified.

Create a test to evaluate a textual answer

In this case, you can use the EvaluateAsync method of the base class AIAgentEvaluation to evaluate the agent and you choose evaluators you want to use among this list:

RelevanceTruthAndCompletenessEvaluator
CoherenceEvaluator
FluencyEvaluator
GroundednessEvaluator

Note

The GroundednessEvaluator evaluator requires an evaluation context to check if the answer is relevant and true.
The context should contain the information needed to check if the answer is correct and not be only a simple expected answer.
This evaluator is useful when the agent uses an external source to answer the question (e.g. using a plugin like RAG's technique or a database).

If your agent use an external source (e.g. using a plugin like RAG's technique), you should use the EvaluateAsync method of the AIAgentEvaluation base class to evaluate the agent.

Here's an example test:

using FluentAssertions;
using GroupeIsa.Neos.Application.XUnit.AI;
using GroupeIsa.Neos.Designer.Application.AIAgents;
using GroupeIsa.Neos.Designer.Domain.Persistence;
using GroupeIsa.Neos.Shared.Linq;
using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.Quality;
using Xunit;
using Xunit.Abstractions;

namespace GroupeIsa.Neos.Designer.Application.Tests.AIAgentEvaluations
{
    /// <inheritdoc/>
    public class EntityExplorerTests : AIAgentEvaluation
    {
        private readonly Mock<IProductRepository>  _productRepositoryMock;
        private readonly Mock<IOrderDetailRepository>  _orderDetailMock;

        public EntityExplorerTests(ITestOutputHelper output, IServiceProvider serviceProvider, Mock<IProductRepository> productRepositoryMock, Mock<IOrderDetailRepository> orderDetailMock)
            : base(output, serviceProvider)
        {
            _productRepositoryMock = productRepositoryMock;
            _orderDetailMock = orderDetailMock;
        }

        [Fact]
        public async Task EntityExplorerAgent_ShouldAnswerTo_UIViewMethodDefinitionLocation()
        {
            // Arrange
            _productRepositoryMock
                .Setup(x => x.GetQuery())
                .Returns(new[] { new Product { Id = 78, Name = "Original Frankfurter grüne Soße" }}.AsQueryable());
            _orderDetailMock
                .Setup(x => x.GetQuery())
                .Returns(new[] { new OrderDetails { ProductId = 10 }}.AsQueryable());

            AIEvaluationOptions options = new AIEvaluationOptions("Which products were never ordered ?")
                .WithEvaluator(AIEvaluator.Groundedness)
                .WithContext("""The products that are never ordered is:
                                    - Product ID: 78, Name: Original Frankfurter grüne Soße
                                    _ Product ID: 10, Name: Ikura""");

            // Act
            EvaluationResult evalResult = await EvaluateAsync<EntityExplorerAgent>(options, CancellationToken.None);

            // Assert
            NumericMetric groundedness = evalResult.Get<NumericMetric>(GroundednessEvaluator.GroundednessMetricName);
            groundedness.Interpretation!.Failed.Should().BeFalse(groundedness.Reason);
        }
    }
}

The question to ask to the agent is defined in the AIEvaluationOptions object.

The response format is text
Product ID: 78 is determined by executing a query in a database in a semantic function

Then you must call the EvaluateAsync method to evaluate the agent.
And you must use the GroundednessEvaluator evaluator to check if the answer is relevant, as this evaluator is the only one that can check the relevance of the response based on the expected response.

In evaluation tests the services are configured with their real implementation. So the repositories use the real implementation to query the database configured in the cluster YAML file.
You can replace the real implementation with a mock implementation if you want to test the agent with a specific data set or to verify a call to a specific method of a particular service.

To do this, in the startup class of the test project, you can add the following code to configure the services:

 public partial class Startup
 {
     /// <summary>
     /// Configures the services.
     /// </summary>
     /// <param name="services">Services.</param>
     public void ConfigureServices(IServiceCollection services)
     {
         _ = InternalConfigureServices(services);

         // Replace the real implementation with a mock
         var productRepositoryMock = new Mock<IProductRepository>();
         services.AddSingleton(productRepositoryMock);
         services.Replace(ServiceDescriptor.Scoped(sp => productRepositoryMock.Object));

         var orderDetailRepositoryMock = new Mock<IOrderDetailRepository>();
         services.AddSingleton(orderDetailRepositoryMock);
         services.Replace(ServiceDescriptor.Scoped(sp => orderDetailRepositoryMock.Object));
     }
 }

The service IProductRepository is replaced by a mock implementation and the Mock<IProductRepository> is configured as singleton in the service collection so it can be retrieved in the test class constructor.

Create a test to evaluate an exact answer

In this case, you should use the PromptAsync method of the AIAgentEvaluation base class to evaluate the agent and check if the response is correct like a classic unit test.

Here's an example test:

[Theory]
[InlineData("Order of March 27, 2025", "OrderView", null, "en", "OrderDate eq 2025-03-27")]
[InlineData("Order with one detail having the item named 'Book'", "OrderView", null, "en", "OrderDetails/any(od: contains(toupper(od/Product/ProductName), 'BOOK'))")]
public async Task ShouldCreate_ODataFilter(string question, string entityViewName, string? uiViewName, string language, string expectedODataFilter)
{
    // Arrange
    AIEvaluationOptions options = new AIEvaluationOptions(
        AiOdataQueryGenerator.GenerateFullPrompt(new EntityViewTextualContext(), entityViewName, question, language, uiViewName),
        expectedODataFilter)
        .WithPromptOptions(new PromptOptions().WithResponseFormat(ResponseFormat.Json));

    // Act
    string answer = await PromptAsync<ODataQueryGeneratorAgent>(options, CancellationToken.None);

    // Assert
    var json = JsonDocument.Parse(answer);
    json.RootElement.GetProperty("filter").GetString().Should().Be(expectedODataFilter);
}

The agent transform a filter in natural language into an OData filter, the expected answer is a JSON response with the filter in the filter property.

How to get the LLM answer

To get the LLM answer, you can use the Answer property of the AIEvaluationBase class. This property contains the current answer of the LLM model.

The answer and the expected answer are also logged in the xUnit output.

Generate reports

First you need to install the dotnet tool :

dotnet tool install --local Microsoft.Extensions.AI.Evaluation.Console --version 9.5.0

Then you can generate the reports using the following command in the root of your cluster after running the tests:

dotnet tool run aieval report --path c:\temp --output report.html

Upgrade your test project

After upgrading Neos, you may need to upgrade your existing test projects.
This is necessary to update the test project to the new version of Neos.
This command updates the version of the Microsoft.Extensions.AI.Evaluation library to the version used in the new version of Neos and updates the startup.generated.cs file if necessary.
You can do this by running the following command in the root of your clusters:

neos upgrade-ai-test

Table of Contents

Evaluation of AI Agents (Preview)

Warning

Getting Started

Evaluators

Warning

Create the xUnit test project

Note

Create a test to evaluate a textual answer

Note

Create a test to evaluate an exact answer

How to get the LLM answer

Generate reports

Upgrade your test project