LangSmith MCP server

What the model can do

LangSmith tracing and evaluation: projects with run counts, latency, cost and error rate, runs in a project (status, errors, tokens, cost, short previews), datasets and examples, feedback and annotation queues. Can add, edit and delete feedback, datasets, examples and tracing projects, create and edit annotation queues and put runs in them, and delete named traces.

Reads

• langsmith_list_projectsLangSmith tracing projects, most recently active first: id, name, run count, error rate, p50 and p99 latency in seconds, tokens and cost.
• langsmith_list_runsRuns in one LangSmith project, newest first by default: name, type (llm, chain, tool…), status, error, latency, tokens, cost, tags and feedback.
• langsmith_list_datasetsLangSmith datasets used for evaluation: id, name, description, type, example count, how many experiments ran on it, created…
• langsmith_list_examplesExamples in one LangSmith dataset, most recent first: id, name, created and modified, metadata…
• langsmith_list_feedbackFeedback recorded on LangSmith runs: key, score or value, comment (cut to 300 characters), source and when.
• langsmith_list_annotation_queuesLangSmith annotation queues, where people review runs: id, name, description, how many runs it holds, reviewers per run and the default dataset.

Acts, when you ask

These change something in LangSmith. The model is told to use them only when you ask for exactly that, and each one is granted by name: per team on a shared connection, by the owner on a personal one.

• langsmith_create_feedbackRecords feedback on one LangSmith run: a key (such as correctness or helpfulness) with a score, a value or a comment.
• langsmith_create_datasetCreates a new, empty LangSmith dataset (key-value type) for evaluation.
• langsmith_create_exampleAdds one example (inputs and, optionally, reference outputs) to a LangSmith dataset.
• langsmith_update_exampleEdits one example in a LangSmith dataset.
• langsmith_update_datasetRenames a LangSmith dataset or changes its description.
• langsmith_update_feedbackChanges one LangSmith feedback record's score, value, comment or correction; fields left out are kept.
• langsmith_create_annotation_queueCreates a LangSmith annotation queue for people to review runs, with optional instructions and rubric (the feedback keys reviewers score).
• langsmith_update_annotation_queueEdits a LangSmith annotation queue: name, description, default dataset, reviewers per run, who completes a run (any reviewer, or the assigned ones)…
• langsmith_add_runs_to_annotation_queuePuts named runs (at most 100) into a LangSmith annotation queue, where its reviewers see them to review.
• langsmith_create_projectCreates a LangSmith tracing project (to hold an application's runs or an experiment) with a name and optional description.
• langsmith_update_projectRenames a LangSmith tracing project or changes its description.
• langsmith_delete_exampleDeletes one example from a LangSmith dataset, permanently.
• langsmith_delete_feedbackDeletes one LangSmith feedback record, permanently.
• langsmith_delete_datasetDeletes a LangSmith dataset and every example in it, permanently.
• langsmith_delete_projectDeletes a LangSmith tracing project and all its runs and traces, permanently.
• langsmith_delete_tracesDeletes named traces (at most 100, by trace id) and all their runs from one LangSmith project, permanently.

Connect it

First connect LangSmith on Dashboard → Connections, then add Memtro to your client once. The LangSmith tools appear alongside the tools of everything else you have connected; nothing is configured per connector.

claude plugin marketplace add memtro-org/claude-memtro
claude plugin install memtro@memtro --config api_key=sk-memtro-...

{
  "mcpServers": {
    "memtro": {
      "url": "https://platform.memtro.com/api/mcp",
      "headers": {
        "Authorization": "Bearer sk-memtro-..."
      }
    }
  }
}

{
  "mcp": {
    "memtro": {
      "type": "remote",
      "url": "https://platform.memtro.com/api/mcp",
      "headers": {
        "Authorization": "Bearer sk-memtro-..."
      }
    }
  }
}

Who may use it

The key identifies the person, and the server offers that person the tools they may use everywhere else in Memtro. A shared LangSmith connection is used by the teams an administrator granted it to, tool by tool, on the Permissions page; a personal connection is used by its owner alone, and its tools that act stay off until the owner allows them. See Teams and permissions and Connectors.

Ask it like this

• › Which agent runs failed today?
• › What are the p99 latency and cost on the support-bot project?
• › What feedback has the correctness key had this week?
• › How many examples are in the refund-questions dataset?
• › Put yesterday's five failed refund runs into the review queue.

Questions

Is this an official LangSmith MCP server?

No. It is Memtro's MCP server with Memtro's LangSmith connector behind it: the connector reads LangSmith through its API with the credentials you connect on Dashboard → Connections, and LangSmith publishes nothing here. The same endpoint carries every other system you connect and Memtro's memory tools.

What does it cost?

Nothing on top of Memtro. One price per organisation by its number of users, from $50 a month for up to 10 users after a 7-day free trial, with every connector, the MCP server, jobs and memory included. You bring your own model keys.

Can it change anything in LangSmith?

Only when you ask, and only what it has been granted: add feedback (a key with a score, value or comment) to a LangSmith run; create a new LangSmith dataset; add an example to a LangSmith dataset; edit a LangSmith dataset example's inputs, reference outputs, metadata or split; rename or re-describe a LangSmith dataset; change the score, value, comment or correction of LangSmith feedback; create a LangSmith annotation queue for people to review runs; edit a LangSmith annotation queue's name, rubric or reviewer settings; put named runs into a LangSmith annotation queue for review; create a LangSmith tracing project; rename or re-describe a LangSmith tracing project; delete a LangSmith dataset example permanently; delete a LangSmith feedback record permanently; delete a LangSmith dataset and all its examples permanently; delete a LangSmith tracing project and all its runs permanently; delete named traces from a LangSmith project permanently. Every other tool reads, searches or lists. An administrator grants each acting tool by name per team, and a personal connection acts only after its owner allows it.

Which clients work?

Any MCP client that speaks Streamable HTTP: Claude Code, Cursor, Claude Desktop, OpenCode, Codex, Zed, Windsurf, Cline and the Anthropic and OpenAI SDKs. There is no local process to run; the server is remote and authenticated with your key.

Other MCP servers · Engineering and operations

• GitHub MCP server
• AWS CloudWatch MCP server
• Sentry MCP server
• ServiceNow MCP server
• Okta MCP server
• Twilio MCP server