Cost control: a monthly token allowance for each modelSee how it works
memtro
Start for free

Braintrust + Memtro

Did last night's eval regress, and which production answers scored low?

Braintrust connects Memtro to your LLM evaluations. It reads experiments with average scores and how they moved against a baseline, a project's recent logs with inputs and outputs cut to 300 characters, and projects and dataset names, never dataset rows. When asked, and once an administrator has granted them by name, it can comment on events, create projects, experiments, datasets, prompts, scores and tags, add examples or log events, and run a saved prompt or scorer, which spends model tokens. Deleting a project or experiment is permanent, and saving a prompt version can change what production sends.

Connect BraintrustStart for freeReference
In Memtro

What you can do with Braintrust in Memtro

Ask in plain English from chat, your editor, a scheduled job or your own agent. The model picks the right Braintrust tool, reads what it needs, and answers with the source cited.

What it does

  1. 01Lists a project's experiments with commit, dataset and tags, and on request their average scores and change against a baseline.
  2. 02Summarises one experiment's scores and its duration, tokens and cost against a comparison experiment, with counts of improvements and regressions.
  3. 03Reads a project's recent traces with model, input, output, expected and error cut to 300 characters, plus scores and tokens, and lists projects and datasets by name.
  4. 04When asked and granted, comments on a log or experiment event, creates or renames projects, experiments and datasets, adds dataset examples, writes log events, and adds review scores and tags.
  5. 05When asked and granted, saves a chat prompt or a new version that code loading it by slug picks up, and runs a saved prompt, scorer or tool, spending model tokens on the organisation's provider keys.
  6. 06When asked and granted, deletes a project with everything in it or an experiment and its results, both permanent, or deletes a dataset, which an administrator can restore in Braintrust.

Acts when asked

When you ask it to

These tools change something in Braintrust. The model is told to use them only when you ask for exactly that, and an administrator grants each one by name to the teams that may use it; a personal connection acts only after its owner allows it.

  1. 01Add a comment to a logged or experiment event
  2. 02Create a project
  3. 03Rename a project or change its description
  4. 04Delete a project with everything in it
  5. 05Create an experiment, linked to a dataset or baseline
  6. 06Rename an experiment or change its description, tags, baseline or dataset
  7. 07Delete an experiment and its results
  8. 08Create a dataset
  9. 09Add examples to a dataset
  10. 10Write events into a project's logs
  11. 11Save a chat prompt, or a new version of one, under a slug
  12. 12Run a saved prompt, scorer or tool on an input, spending model tokens
  13. 13Add a slider, categorical or free-form review score to a project
  14. 14Add a tag to a project
  15. 15Delete a dataset
In practice

Ask it like you would a colleague

Once Braintrust is connected, these are the kinds of things people ask. Plain questions, no syntax. If the answer lives partly in another system you have connected, Memtro looks there too.

Braintrust + Memtro

  • Did last night's eval regress?

  • Show recent low-scoring answers in the support bot project.

  • How did cost and tokens move in the latest experiment against its baseline?

  • Add these three failed support answers to the refunds dataset with the expected replies.

  • Add a comment to that log event saying retrieval missed the refunds policy.

01

Checking an overnight eval

Dan Fletcher, ML engineer at BigCo, asked whether last night's eval regressed. Memtro summarised the experiment against its baseline, showing which scores fell and how many cases regressed.

02

Turning failures into tests

Sarah Chen, AI product lead at Northwind, asked for today's low-scoring answers, then asked Memtro to add three of them to the refunds dataset with the replies she wrote. Memtro added the three examples, and every later eval run on that dataset used them.

Teams

Where it fits

The teams that lean on Braintrust most, a question each of them asks, and what they connect beside it.

Better together

Works with

A record in Braintrust is rarely the whole story. Connect the systems that hold the rest and Memtro checks them together, then remembers what it learned so the next question starts further along.

Also in Engineering and operations

Using something else for the same job, or both? Each connector has its own tools; connect as many as you like.

Setup

How to connect

Connect withAPI key and optional data plane region
  1. 1

    In braintrust.dev, open Settings → API keys as a dedicated user. The key has no scopes and acts as that user, so give the user read access to the projects Memtro should see, and more only for the actions an administrator will grant.

  2. 2

    Create a key named for Memtro and copy it, as it is shown once.

  3. 3

    In Memtro, open Connections → Braintrust, paste the API key, and set Data plane region to eu if your organisation uses it.

  4. 4

    Ask: did last night's eval regress?

Also works from

Chat in the dashboard, the OpenAI-compatible API, scheduled jobs, and any MCP client: Claude Code, OpenCode, Cursor, Claude Desktop. The Braintrust tools are the same everywhere.

Set it up as an MCP serverClients and agents
FAQ

Questions

Can Memtro change anything in Braintrust?+

Only when you ask, and only what it has been granted: add a comment to a logged or experiment event; create a project; rename a project or change its description; delete a project with everything in it; create an experiment, linked to a dataset or baseline; rename an experiment or change its description, tags, baseline or dataset; delete an experiment and its results; create a dataset; add examples to a dataset; write events into a project's logs; save a chat prompt, or a new version of one, under a slug; run a saved prompt, scorer or tool on an input, spending model tokens; add a slider, categorical or free-form review score to a project; add a tag to a project; delete a dataset. Every other tool reads, searches or lists. The Connections page labels the tools that act, an administrator grants each one by name to the teams that may use it, and a personal connection acts only after its owner allows it.

Who can use a Braintrust connection?+

A personal connection serves only your own requests. A shared connection serves the organisation: everyone, or only the teams an admin chooses. Admins can always add shared connections; other members need the permission.

Does Memtro store my Braintrust data?+

Credentials are encrypted at rest. Tool results are cached briefly so agent loops do not repeat identical calls, and the facts the model extracts are saved to your organisation's memory with provenance (system, record and link) so they can be checked and superseded.

How do I connect Braintrust?+

API key and optional data plane region. In braintrust.dev, open Settings → API keys as a dedicated user. The key has no scopes and acts as that user, so give the user read access to the projects Memtro should see, and more only for the actions an administrator will grant.

Connect Braintrust in a few minutes

Seven-day free trial, then one monthly price per organisation. Connectors, jobs and memory are never metered.

Start for freePricing

Where agents remember your company

Sign in, add a provider key, create an API key and point your tools at Memtro. It takes about five minutes.

Start for freeGetting started

7-day free trial · card on file, nothing charged until it ends · cancel any time