Run experiments in code

Running your evaluation as code with the Braintrust SDK gives you full control over the task, data, and scorers, and lets you version it alongside your application. Iterate locally, then wire the same code into CI/CD.

Run locally

Run evaluation code locally to create an experiment in Braintrust and return summary metrics, including a direct link to your experiment. See Interpret results for how to read it.

Install the SDK and dependencies:

# pnpm
pnpm add braintrust openai autoevals
# npm
npm install braintrust openai autoevals

Create the eval code:

import { Eval, initDataset } from "braintrust";
import { Factuality } from "autoevals";

Eval("My Project", {
  experimentName: "My experiment",
  data: initDataset("My Project", { dataset: "My dataset" }),
  task: async (input) => {
    // Your LLM call here
    return await callModel(input);
  },
  scores: [Factuality],
  metadata: {
    model: "gpt-5-mini",
  },
});

Run your evaluation with the bt eval CLI:

bt eval my_eval.eval.ts

Use --watch to re-run automatically when files change:

bt eval --watch my_eval.eval.ts

Benefits of using the CLI:

Automatic .env loading — reads .env.development.local, .env.local, .env.development, and .env
Multi-file support — pass multiple files or directories: bt eval [file or directory] .... Running bt eval with no arguments runs all eval files in the current directory.
TypeScript transpilation — no build step required; the CLI handles it

Install the SDK and dependencies:

pip install braintrust openai autoevals

Create the eval code:

from braintrust import Eval, init_dataset
from autoevals import Factuality

Eval(
    "My project",
    experiment_name="My experiment",
    data=init_dataset(project="My project", name="My dataset"),
    task=lambda input: call_model(input),  # Your LLM call here
    scores=[Factuality],
    metadata={
        "model": "gpt-5-mini",
    },
)

Run your evaluation with the bt eval CLI:

bt eval my_eval.py

Use --watch to re-run automatically when files change:

bt eval --watch my_eval.py

Benefits of using the CLI:

Automatic .env loading — reads .env.development.local, .env.local, .env.development, and .env
Multi-file support — pass multiple files or directories: bt eval [file or directory] .... Running bt eval with no arguments runs all eval files in the current directory.
TypeScript transpilation — no build step required; the CLI handles it

Install the SDK and dependencies:

go get github.com/braintrustdata/braintrust-sdk-go
go get github.com/openai/openai-go

Create the eval code:

package main

import (
	"context"
	"log"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/sdk/trace"

	"github.com/braintrustdata/braintrust-sdk-go"
	"github.com/braintrustdata/braintrust-sdk-go/eval"
)

func callModel(input string) string {
	// Your LLM call implementation here
	return "model output"
}

func main() {
	ctx := context.Background()

	tp := trace.NewTracerProvider()
	defer tp.Shutdown(ctx)
	otel.SetTracerProvider(tp)

	client, err := braintrust.New(tp)
	if err != nil {
		log.Fatal(err)
	}

	evaluator := braintrust.NewEvaluator[string, string](client)

	_, err = evaluator.Run(ctx, eval.Opts[string, string]{
		Experiment: "My project",
		Dataset: eval.NewDataset([]eval.Case[string, string]{
			{Input: "example input", Expected: "example expected"},
		}),
		Task: eval.T(func(ctx context.Context, input string) (string, error) {
			return callModel(input), nil // Your LLM call here
		}),
		Scorers: []eval.Scorer[string, string]{
			eval.NewScorer("exact-match", func(ctx context.Context, r eval.TaskResult[string, string]) (eval.Scores, error) {
				score := 0.0
				if r.Output == r.Expected {
					score = 1.0
				}
				return eval.S(score), nil
			}),
		},
		Metadata: map[string]any{
			"model": "gpt-5-mini",
		},
	})
	if err != nil {
		log.Fatal(err)
	}
}

Run your evaluation:

go run my_eval.go

Install the SDK and dependencies:

# Add to your Gemfile:
gem "braintrust"
gem "openai"

bundle install

Create the eval code:

require "braintrust"

Braintrust.init

Braintrust::Eval.run(
  project: "My project",
  cases: [
    {input: "example input", expected: "example expected"},
  ],
  task: ->(input:) { call_model(input) },  # Your LLM call here
  scorers: [
    Braintrust::Scorer.new("exact_match") { |expected:, output:| output == expected ? 1.0 : 0.0 }
  ],
  metadata: {model: "gpt-5-mini"}
)

OpenTelemetry.tracer_provider.shutdown

Run your evaluation:

ruby my_eval.rb

Install the SDK and dependencies:

# Add to build.gradle dependencies{} block:
implementation 'dev.braintrust:braintrust-sdk-java:<version>'
implementation 'com.openai:openai-java-sdk:<version>'

Create the eval code:

import dev.braintrust.Braintrust;
import dev.braintrust.eval.DatasetCase;
import dev.braintrust.eval.Scorer;

class Main {
  static String callModel(String input) {
    // Your LLM call implementation here
    return "model output";
  }

  public static void main(String... args) {
    var braintrust = Braintrust.get();
    var openTelemetry = braintrust.openTelemetryCreate();

    var eval = braintrust.<String, String>evalBuilder()
        .name("My project")
        .cases(DatasetCase.of("example input", "example expected"))
        .taskFunction(input -> callModel(input)) // Your LLM call here
        .scorers(
            Scorer.of("exact_match", (expected, actual) -> expected.equals(actual) ? 1.0 : 0.0)
        )
        .metadata(java.util.Map.of(
            "model", "gpt-5-mini"
        ))
        .build();

    var result = eval.run();
    System.out.println(result.createReportString());
  }
}

Run your evaluation:

javac -cp ".:*" MyEval.java
java -cp ".:*" MyEval

Install the SDK and dependencies:

dotnet add package Braintrust.Sdk
dotnet add package Braintrust.Sdk.OpenAI
dotnet add package OpenAI

Create the eval code:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Braintrust.Sdk;
using Braintrust.Sdk.Eval;

class Program
{
    static string CallModel(string input)
    {
        // Your LLM call implementation here
        return "model output";
    }

    static async Task Main(string[] args)
    {
        var braintrust = Braintrust.Sdk.Braintrust.Get();

        var eval = await braintrust
            .EvalBuilder<string, string>()
            .Name("My Project")
            .Cases(
                new DatasetCase<string, string>("example input", "example expected")
            )
            .TaskFunction(input => CallModel(input)) // Your LLM call here
            .Scorers(
                new FunctionScorer<string, string>("exact_match", (expected, actual) =>
                    actual == expected ? 1.0 : 0.0)
            )
            .BuildAsync();

        var result = await eval.RunAsync();
        Console.WriteLine(result.CreateReportString());
    }
}

Run your evaluation:

dotnet run

You can pass a parameters option to make configuration values (like model choice, temperature, or prompts) editable in the playground without changing code. Define parameters inline or use loadParameters() to reference saved configurations. See Write parameters and Test complex agents for details.

Configure experiments

Customize experiment behavior with options:

Eval("My Project", {
  data: myDataset,
  task: myTask,
  scores: [Factuality],

  // Experiment name
  experiment: "gpt-5-mini-experiment",

  // Metadata for filtering/analysis
  metadata: {
    model: "gpt-5-mini",
    prompt_version: "v2",
  },

  // Maximum concurrency
  maxConcurrency: 10,

  // Trial count for averaging
  trialCount: 3,
});

Eval(
    "My Project",
    data=my_dataset,
    task=my_task,
    scores=[Factuality],

    # Experiment name
    experiment="gpt-5-mini-experiment",

    # Metadata for filtering/analysis
    metadata={
        "model": "gpt-5-mini",
        "prompt_version": "v2",
    },

    # Maximum concurrency
    max_concurrency=10,

    # Trial count for averaging
    trial_count=3,
)

package main

import (
	"context"
	"log"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/sdk/trace"

	"github.com/braintrustdata/braintrust-sdk-go"
	"github.com/braintrustdata/braintrust-sdk-go/eval"
)

func main() {
	ctx := context.Background()

	tp := trace.NewTracerProvider()
	defer tp.Shutdown(ctx)
	otel.SetTracerProvider(tp)

	client, err := braintrust.New(tp)
	if err != nil {
		log.Fatal(err)
	}

	evaluator := braintrust.NewEvaluator[string, string](client)

	// Example variables (define your own)
	myDataset := eval.NewDataset([]eval.Case[string, string]{
		{Input: "example input", Expected: "example expected"},
	})
	myTask := eval.T(func(ctx context.Context, input string) (string, error) {
		return "model output", nil
	})
	myScorers := []eval.Scorer[string, string]{
		eval.NewScorer("exact-match", func(ctx context.Context, r eval.TaskResult[string, string]) (eval.Scores, error) {
			score := 0.0
			if r.Output == r.Expected {
				score = 1.0
			}
			return eval.S(score), nil
		}),
	}

	_, err = evaluator.Run(ctx, eval.Opts[string, string]{
		Experiment: "gpt-5-mini-experiment",
		Dataset:    myDataset,
		Task:       myTask,
		Scorers:    myScorers,
		Metadata: map[string]any{
			"model":          "gpt-5-mini",
			"prompt_version": "v2",
		},
	})
	if err != nil {
		log.Fatal(err)
	}
}

Braintrust::Eval.run(
  project: "My Project",
  experiment: "gpt-5-mini-experiment",
  cases: my_dataset,
  task: my_task,
  scorers: my_scorers,
  metadata: {model: "gpt-5-mini", prompt_version: "v2"},
  max_concurrency: 10,
  trial_count: 3
)

import dev.braintrust.Braintrust;
import dev.braintrust.eval.DatasetCase;
import dev.braintrust.eval.Scorer;
import java.util.function.Function;

class Main {
  public static void main(String... args) {
    var braintrust = Braintrust.get();
    var openTelemetry = braintrust.openTelemetryCreate();

    // Example variables (define your own)
    var myDataset = new DatasetCase[]{
        DatasetCase.of("example input", "example expected")
    };
    Function<String, String> myTask = input -> "model output";
    var myScorers = new Scorer[]{
        Scorer.of("exact_match", (expected, actual) -> expected.equals(actual) ? 1.0 : 0.0)
    };

    var result = braintrust.<String, String>evalBuilder()
        .name("gpt-5-mini-experiment")
        .cases(myDataset)
        .taskFunction(myTask)
        .scorers(myScorers)
        .metadata(java.util.Map.of(
            "model", "gpt-5-mini",
            "prompt_version", "v2"
        ))
        .build()
        .run();

    System.out.println(result.createReportString());
  }
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Braintrust.Sdk;
using Braintrust.Sdk.Eval;

class Program
{
    static async Task Main(string[] args)
    {
        var braintrust = Braintrust.Sdk.Braintrust.Get();

        // Example variables (define your own)
        var myDataset = new[] {
            new DatasetCase<string, string>("example input", "example expected")
        };
        Func<string, string> myTask = input => "model output";
        var myScorers = new[] {
            new FunctionScorer<string, string>("exact_match", (expected, actual) =>
                actual == expected ? 1.0 : 0.0)
        };

        var eval = await braintrust
            .EvalBuilder<string, string>()
            .Name("gpt-5-mini-experiment")
            .Cases(myDataset)
            .TaskFunction(myTask)
            .Scorers(myScorers)
            .BuildAsync();

        var result = await eval.RunAsync();
        Console.WriteLine(result.CreateReportString());
    }
}

In the TypeScript, Python, and Ruby SDKs, you can set each inline eval case’s origin field to link it back to its source dataset row. This populates the dataset performance view, which only shows experiments whose eval traces set origin.

These examples show common options. For the complete set of Eval() options, see the SDK reference for your language.

Next steps

Interpret results from your experiments
Compare experiments to measure improvements
Advanced eval techniques like trials, hill climbing, attachments, and tracing
Run experiments in the UI without writing code
Run experiments in CI/CD to catch regressions automatically

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

Run experiments in code

Run locally

Configure experiments

Next steps

​Run locally

​Configure experiments

​Next steps

Run locally

Configure experiments

Next steps