LangSmith logo

LangSmith + Memtro

Which agent runs failed today, and what are latency and cost doing?

LangSmith traces and evaluates your LLM applications. Memtro reads tracing projects with run count, error rate, p50 and p99 latency, tokens and cost, the runs in a project, datasets with their examples, and feedback on runs. Prompts and outputs are not returned, only short previews when asked. When asked and once an administrator has granted the action by name, it can record and edit feedback, create and edit datasets, examples, projects and annotation queues, and put runs into a queue for review. Deleting a dataset, example, feedback, project or named traces is permanent. A tracing-only key cannot read.

In Memtro

What you can do with LangSmith in Memtro

Ask in plain English from chat, your editor, a scheduled job or your own agent. The model picks the right LangSmith tool, reads what it needs, and answers with the source cited.

What it does

  1. 01Lists tracing projects with run count, error rate, p50 and p99 latency, tokens and cost
  2. 02Lists runs in a project with type, status, error, latency, tokens, cost and feedback, filtered to errors or whole traces
  3. 03Reads datasets with example and experiment counts, examples with inputs and reference outputs cut to 300 characters, and feedback by key
  4. 04When asked and granted, records or edits feedback on a run, and creates or edits datasets and their examples. Everyone in the workspace sees them, and nobody is notified.
  5. 05When asked and granted, creates or renames tracing projects, creates and edits annotation queues, and puts up to 100 named runs into a queue for its reviewers
  6. 06When asked and granted, permanently deletes a dataset with its examples, an example, a feedback record, a project with all its runs, or named traces

Acts when asked

When you ask it to

These tools change something in LangSmith. The model is told to use them only when you ask for exactly that, and an administrator grants each one by name to the teams that may use it; a personal connection acts only after its owner allows it.

  1. 01Add feedback (a key with a score, value or comment) to a LangSmith run
  2. 02Create a new LangSmith dataset
  3. 03Add an example to a LangSmith dataset
  4. 04Edit a LangSmith dataset example's inputs, reference outputs, metadata or split
  5. 05Rename or re-describe a LangSmith dataset
  6. 06Change the score, value, comment or correction of LangSmith feedback
  7. 07Create a LangSmith annotation queue for people to review runs
  8. 08Edit a LangSmith annotation queue's name, rubric or reviewer settings
  9. 09Put named runs into a LangSmith annotation queue for review
  10. 10Create a LangSmith tracing project
  11. 11Rename or re-describe a LangSmith tracing project
  12. 12Delete a LangSmith dataset example permanently
  13. 13Delete a LangSmith feedback record permanently
  14. 14Delete a LangSmith dataset and all its examples permanently
  15. 15Delete a LangSmith tracing project and all its runs permanently
  16. 16Delete named traces from a LangSmith project permanently

Events

When it happens

Turn events on in the connection's Events tab on the Connections page, then add the address it shows as a webhook in LangSmith, choosing the same kinds; the tab shows LangSmith's own steps. Each event can start a job, with the event as its input. LangSmith sends a secret with every delivery, and one without it is refused.

Events and webhooks
  • 01An automation rule matched a run
  • 02An automation rule matched an idle thread
In practice

Ask it like you would a colleague

Once LangSmith is connected, these are the kinds of things people ask. Plain questions, no syntax. If the answer lives partly in another system you have connected, Memtro looks there too.

LangSmith logoLangSmith + Memtro
  • Which agent runs failed today?

  • What are the p99 latency and cost on the support-bot project?

  • What feedback has the correctness key had this week?

  • How many examples are in the refund-questions dataset?

  • Put yesterday's five failed refund runs into the review queue.

01

Morning error check

Tom Reilly, ML Engineer at Northwind, asked which agent runs had failed overnight. Memtro listed the error runs in the agent's project with their errors, latency and tokens, beside the project's error rate.

02

Turning a failure into a test

Priya Sharma, AI Lead at Acme Ltd, asked Memtro to mark a run as incorrect and add its input to the regression dataset. Memtro recorded correctness feedback on the run and added the example. Everyone in the workspace could see both, and the next experiment on the dataset included it.

Teams

Where it fits

The teams that lean on LangSmith most, a question each of them asks, and what they connect beside it.

Better together

Works with

A record in LangSmith is rarely the whole story. Connect the systems that hold the rest and Memtro checks them together, then remembers what it learned so the next question starts further along.

Also in Engineering and operations

Using something else for the same job, or both? Each connector has its own tools; connect as many as you like.

Setup

How to connect

Connect withAPI key
  1. 1

    In LangSmith, open Settings → API Keys → Create API Key and choose a Service Key with a role that can read.

  2. 2

    Give the role write access only if you want the actions, each of which still needs an administrator's grant in Memtro. Copy the key, as it is shown once.

  3. 3

    In Memtro, open Connections → LangSmith and paste the API key. Set Region to eu for eu.smith.langchain.com, and the Workspace ID if the key spans several.

  4. 4

    Ask: Which agent runs failed today?

Also works from

Chat in the dashboard, the OpenAI-compatible API, scheduled jobs, and any MCP client: Claude Code, OpenCode, Cursor, Claude Desktop. The LangSmith tools are the same everywhere.

FAQ

Questions

Can Memtro change anything in LangSmith?+

Only when you ask, and only what it has been granted: add feedback (a key with a score, value or comment) to a LangSmith run; create a new LangSmith dataset; add an example to a LangSmith dataset; edit a LangSmith dataset example's inputs, reference outputs, metadata or split; rename or re-describe a LangSmith dataset; change the score, value, comment or correction of LangSmith feedback; create a LangSmith annotation queue for people to review runs; edit a LangSmith annotation queue's name, rubric or reviewer settings; put named runs into a LangSmith annotation queue for review; create a LangSmith tracing project; rename or re-describe a LangSmith tracing project; delete a LangSmith dataset example permanently; delete a LangSmith feedback record permanently; delete a LangSmith dataset and all its examples permanently; delete a LangSmith tracing project and all its runs permanently; delete named traces from a LangSmith project permanently. Every other tool reads, searches or lists. The Connections page labels the tools that act, an administrator grants each one by name to the teams that may use it, and a personal connection acts only after its owner allows it.

Who can use a LangSmith connection?+

A personal connection serves only your own requests. A shared connection serves the organisation: everyone, or only the teams an admin chooses. Admins can always add shared connections; other members need the permission.

Does Memtro store my LangSmith data?+

Credentials are encrypted at rest. Tool results are cached briefly so agent loops do not repeat identical calls, and the facts the model extracts are saved to your organisation's memory with provenance (system, record and link) so they can be checked and superseded.

How do I connect LangSmith?+

API key. In LangSmith, open Settings → API Keys → Create API Key and choose a Service Key with a role that can read.

Can LangSmith start a job?+

Yes. Turn events on in the connection's Events tab on the Connections page, then add the address it shows as a webhook in LangSmith, choosing the same kinds; the tab shows LangSmith's own steps. A job set to start "When an event arrives" then runs once for each matching event, with the event as its input. The run treats the event as outside data, never as instructions, and anything the run remembers is kept as unverified.

Connect LangSmith in a few minutes

Seven-day free trial, then one monthly price per organisation. Connectors, jobs and memory are never metered.

Where agents remember your company

Sign in, add a provider key, create an API key and point your tools at Memtro. It takes about five minutes.

7-day free trial · card on file, nothing charged until it ends · cancel any time