> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Run experiments in code
> Run evaluations with the Braintrust SDK, from a local script or the bt eval CLI, and capture each run as an immutable experiment snapshot.
Running your evaluation as code with the Braintrust SDK gives you full control over the task, data, and scorers, and lets you version it alongside your application. Iterate locally, then wire the same code into [CI/CD](/docs/evaluate/run-in-ci).
## Run locally
Run evaluation code locally to create an experiment in Braintrust and return summary metrics, including a direct link to your experiment. See [Interpret results](/docs/evaluate/interpret-results) for how to read it.
Install the SDK and dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# pnpm
pnpm add braintrust openai autoevals
# npm
npm install braintrust openai autoevals
```
Create the eval code:
```typescript wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import { Eval, initDataset } from "braintrust";
import { Factuality } from "autoevals";
Eval("My Project", {
experimentName: "My experiment",
data: initDataset("My Project", { dataset: "My dataset" }),
task: async (input) => {
// Your LLM call here
return await callModel(input);
},
scores: [Factuality],
metadata: {
model: "gpt-5-mini",
},
});
```
Run your evaluation with the [`bt eval`](/docs/reference/cli/eval) CLI:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval my_eval.eval.ts
```
Use `--watch` to re-run automatically when files change:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval --watch my_eval.eval.ts
```
**Benefits of using the CLI:**
* **Automatic `.env` loading** — reads `.env.development.local`, `.env.local`, `.env.development`, and `.env`
* **Multi-file support** — pass multiple files or directories: `bt eval [file or directory] ...`. Running `bt eval` with no arguments runs all eval files in the current directory.
* **TypeScript transpilation** — no build step required; the CLI handles it
Install the SDK and dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
pip install braintrust openai autoevals
```
Create the eval code:
```python wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from braintrust import Eval, init_dataset
from autoevals import Factuality
Eval(
"My project",
experiment_name="My experiment",
data=init_dataset(project="My project", name="My dataset"),
task=lambda input: call_model(input), # Your LLM call here
scores=[Factuality],
metadata={
"model": "gpt-5-mini",
},
)
```
Run your evaluation with the [`bt eval`](/docs/reference/cli/eval) CLI:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval my_eval.py
```
Use `--watch` to re-run automatically when files change:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval --watch my_eval.py
```
**Benefits of using the CLI:**
* **Automatic `.env` loading** — reads `.env.development.local`, `.env.local`, `.env.development`, and `.env`
* **Multi-file support** — pass multiple files or directories: `bt eval [file or directory] ...`. Running `bt eval` with no arguments runs all eval files in the current directory.
* **TypeScript transpilation** — no build step required; the CLI handles it
Install the SDK and dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
go get github.com/braintrustdata/braintrust-sdk-go
go get github.com/openai/openai-go
```
Create the eval code:
```go wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
package main
import (
"context"
"log"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/sdk/trace"
"github.com/braintrustdata/braintrust-sdk-go"
"github.com/braintrustdata/braintrust-sdk-go/eval"
)
func callModel(input string) string {
// Your LLM call implementation here
return "model output"
}
func main() {
ctx := context.Background()
tp := trace.NewTracerProvider()
defer tp.Shutdown(ctx)
otel.SetTracerProvider(tp)
client, err := braintrust.New(tp)
if err != nil {
log.Fatal(err)
}
evaluator := braintrust.NewEvaluator[string, string](client)
_, err = evaluator.Run(ctx, eval.Opts[string, string]{
Experiment: "My project",
Dataset: eval.NewDataset([]eval.Case[string, string]{
{Input: "example input", Expected: "example expected"},
}),
Task: eval.T(func(ctx context.Context, input string) (string, error) {
return callModel(input), nil // Your LLM call here
}),
Scorers: []eval.Scorer[string, string]{
eval.NewScorer("exact-match", func(ctx context.Context, r eval.TaskResult[string, string]) (eval.Scores, error) {
score := 0.0
if r.Output == r.Expected {
score = 1.0
}
return eval.S(score), nil
}),
},
Metadata: map[string]any{
"model": "gpt-5-mini",
},
})
if err != nil {
log.Fatal(err)
}
}
```
Run your evaluation:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
go run my_eval.go
```
Install the SDK and dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# Add to your Gemfile:
gem "braintrust"
gem "openai"
bundle install
```
Create the eval code:
```ruby wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
require "braintrust"
Braintrust.init
Braintrust::Eval.run(
project: "My project",
cases: [
{input: "example input", expected: "example expected"},
],
task: ->(input:) { call_model(input) }, # Your LLM call here
scorers: [
Braintrust::Scorer.new("exact_match") { |expected:, output:| output == expected ? 1.0 : 0.0 }
],
metadata: {model: "gpt-5-mini"}
)
OpenTelemetry.tracer_provider.shutdown
```
Run your evaluation:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
ruby my_eval.rb
```
Install the SDK and dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# Add to build.gradle dependencies{} block:
implementation 'dev.braintrust:braintrust-sdk-java:'
implementation 'com.openai:openai-java-sdk:'
```
Create the eval code:
```java wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import dev.braintrust.Braintrust;
import dev.braintrust.eval.DatasetCase;
import dev.braintrust.eval.Scorer;
class Main {
static String callModel(String input) {
// Your LLM call implementation here
return "model output";
}
public static void main(String... args) {
var braintrust = Braintrust.get();
var openTelemetry = braintrust.openTelemetryCreate();
var eval = braintrust.evalBuilder()
.name("My project")
.cases(DatasetCase.of("example input", "example expected"))
.taskFunction(input -> callModel(input)) // Your LLM call here
.scorers(
Scorer.of("exact_match", (expected, actual) -> expected.equals(actual) ? 1.0 : 0.0)
)
.metadata(java.util.Map.of(
"model", "gpt-5-mini"
))
.build();
var result = eval.run();
System.out.println(result.createReportString());
}
}
```
Run your evaluation:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
javac -cp ".:*" MyEval.java
java -cp ".:*" MyEval
```
Install the SDK and dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
dotnet add package Braintrust.Sdk
dotnet add package Braintrust.Sdk.OpenAI
dotnet add package OpenAI
```
Create the eval code:
```csharp wrap theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Braintrust.Sdk;
using Braintrust.Sdk.Eval;
class Program
{
static string CallModel(string input)
{
// Your LLM call implementation here
return "model output";
}
static async Task Main(string[] args)
{
var braintrust = Braintrust.Sdk.Braintrust.Get();
var eval = await braintrust
.EvalBuilder()
.Name("My Project")
.Cases(
new DatasetCase("example input", "example expected")
)
.TaskFunction(input => CallModel(input)) // Your LLM call here
.Scorers(
new FunctionScorer("exact_match", (expected, actual) =>
actual == expected ? 1.0 : 0.0)
)
.BuildAsync();
var result = await eval.RunAsync();
Console.WriteLine(result.CreateReportString());
}
}
```
Run your evaluation:
```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
dotnet run
```
You can pass a `parameters` option to make configuration values (like model choice, temperature, or prompts) editable in the playground without changing code. Define parameters inline or use `loadParameters()` to reference saved configurations. See [Write parameters](/docs/evaluate/write-parameters) and [Test complex agents](/docs/evaluate/remote-evals) for details.
Customize experiment behavior with options:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
Eval("My Project", {
data: myDataset,
task: myTask,
scores: [Factuality],
// Experiment name
experiment: "gpt-5-mini-experiment",
// Metadata for filtering/analysis
metadata: {
model: "gpt-5-mini",
prompt_version: "v2",
},
// Maximum concurrency
maxConcurrency: 10,
// Trial count for averaging
trialCount: 3,
});
```
```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
Eval(
"My Project",
data=my_dataset,
task=my_task,
scores=[Factuality],
# Experiment name
experiment="gpt-5-mini-experiment",
# Metadata for filtering/analysis
metadata={
"model": "gpt-5-mini",
"prompt_version": "v2",
},
# Maximum concurrency
max_concurrency=10,
# Trial count for averaging
trial_count=3,
)
```
```go theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
package main
import (
"context"
"log"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/sdk/trace"
"github.com/braintrustdata/braintrust-sdk-go"
"github.com/braintrustdata/braintrust-sdk-go/eval"
)
func main() {
ctx := context.Background()
tp := trace.NewTracerProvider()
defer tp.Shutdown(ctx)
otel.SetTracerProvider(tp)
client, err := braintrust.New(tp)
if err != nil {
log.Fatal(err)
}
evaluator := braintrust.NewEvaluator[string, string](client)
// Example variables (define your own)
myDataset := eval.NewDataset([]eval.Case[string, string]{
{Input: "example input", Expected: "example expected"},
})
myTask := eval.T(func(ctx context.Context, input string) (string, error) {
return "model output", nil
})
myScorers := []eval.Scorer[string, string]{
eval.NewScorer("exact-match", func(ctx context.Context, r eval.TaskResult[string, string]) (eval.Scores, error) {
score := 0.0
if r.Output == r.Expected {
score = 1.0
}
return eval.S(score), nil
}),
}
_, err = evaluator.Run(ctx, eval.Opts[string, string]{
Experiment: "gpt-5-mini-experiment",
Dataset: myDataset,
Task: myTask,
Scorers: myScorers,
Metadata: map[string]any{
"model": "gpt-5-mini",
"prompt_version": "v2",
},
})
if err != nil {
log.Fatal(err)
}
}
```
```ruby theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
Braintrust::Eval.run(
project: "My Project",
experiment: "gpt-5-mini-experiment",
cases: my_dataset,
task: my_task,
scorers: my_scorers,
metadata: {model: "gpt-5-mini", prompt_version: "v2"},
max_concurrency: 10,
trial_count: 3
)
```
```java theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import dev.braintrust.Braintrust;
import dev.braintrust.eval.DatasetCase;
import dev.braintrust.eval.Scorer;
import java.util.function.Function;
class Main {
public static void main(String... args) {
var braintrust = Braintrust.get();
var openTelemetry = braintrust.openTelemetryCreate();
// Example variables (define your own)
var myDataset = new DatasetCase[]{
DatasetCase.of("example input", "example expected")
};
Function myTask = input -> "model output";
var myScorers = new Scorer[]{
Scorer.of("exact_match", (expected, actual) -> expected.equals(actual) ? 1.0 : 0.0)
};
var result = braintrust.evalBuilder()
.name("gpt-5-mini-experiment")
.cases(myDataset)
.taskFunction(myTask)
.scorers(myScorers)
.metadata(java.util.Map.of(
"model", "gpt-5-mini",
"prompt_version", "v2"
))
.build()
.run();
System.out.println(result.createReportString());
}
}
```
```csharp theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Braintrust.Sdk;
using Braintrust.Sdk.Eval;
class Program
{
static async Task Main(string[] args)
{
var braintrust = Braintrust.Sdk.Braintrust.Get();
// Example variables (define your own)
var myDataset = new[] {
new DatasetCase("example input", "example expected")
};
Func myTask = input => "model output";
var myScorers = new[] {
new FunctionScorer("exact_match", (expected, actual) =>
actual == expected ? 1.0 : 0.0)
};
var eval = await braintrust
.EvalBuilder()
.Name("gpt-5-mini-experiment")
.Cases(myDataset)
.TaskFunction(myTask)
.Scorers(myScorers)
.BuildAsync();
var result = await eval.RunAsync();
Console.WriteLine(result.CreateReportString());
}
}
```
In the TypeScript, Python, and Ruby SDKs, you can set each inline eval case's `origin` field to link it back to its source dataset row. This populates the [dataset performance view](/docs/annotate/datasets/track-performance), which only shows experiments whose eval traces set `origin`.
These examples show common options. For the complete set of `Eval()` options, see the [SDK reference](/docs/sdks) for your language.
## Next steps
* [Interpret results](/docs/evaluate/interpret-results) from your experiments
* [Compare experiments](/docs/evaluate/compare-experiments) to measure improvements
* [Advanced eval techniques](/docs/evaluate/advanced-evaluations) like trials, hill climbing, attachments, and tracing
* [Run experiments in the UI](/docs/evaluate/run-in-ui) without writing code
* [Run experiments in CI/CD](/docs/evaluate/run-in-ci) to catch regressions automatically