Evaluation failures reviewed
Maya Okafor, AI product lead at Acme Ltd, asked which traces failed evaluation yesterday. Memtro listed them with their cost, latency and labels. She saw most came from one feature and opened its prompt next.
LangWatch + Memtro
Which traces failed evaluation yesterday, and what did they cost?
LangWatch connects Memtro to your LLM observability project. It reads traces with cost, latency, tokens, labels, evaluation verdicts and feedback, reviewers' annotations, and managed prompts with their versions. It also reads evaluation monitors, automations, experiments, simulation runs and datasets. When asked and granted, it can annotate traces, edit, restore or delete prompts, switch monitors and automations, run experiments, add dataset records and set trace metadata. LangWatch can also tell Memtro about AI Gateway spend, budgets and virtual keys, and a job can start on it. Those events need LangWatch Enterprise, and trace inputs and outputs are never returned.
In Memtro
What you can do with LangWatch in Memtro
Ask in plain English from chat, your editor, a scheduled job or your own agent. The model picks the right LangWatch tool, reads what it needs, and answers with the source cited.
What it does
- 01Traces in a time window with cost, latency, tokens, labels, evaluation verdicts and thumbs feedback.
- 02Reviewers' annotations, and managed prompts with their current version, model and settings.
- 03Online-evaluation monitors, automations, experiments and their runs, agent simulation runs and datasets.
- 04When asked and granted, annotates a trace for everyone on the project, edits or deletes an annotation, and sets metadata or labels on a trace.
- 05When asked and granted, creates, changes, restores or deletes a prompt, which apps fetching it receive straight away, and switches monitors or automations on or off.
- 06With events on, an AI Gateway request finishing, a budget crossing its warning or a budget reaching its cap can start a job.
Acts when asked
When you ask it to
These tools change something in LangWatch. The model is told to use them only when you ask for exactly that, and an administrator grants each one by name to the teams that may use it; a personal connection acts only after its owner allows it.
- 01Annotate a trace with a comment and thumbs up or down
- 02Edit an annotation
- 03Delete an annotation
- 04Create a managed prompt
- 05Change a prompt, saving a new version that apps fetching it receive
- 06Restore a prompt to an earlier version
- 07Delete a prompt
- 08Switch an online-evaluation monitor on or off
- 09Pause or resume an automation (alerts and reports)
- 10Run an experiment (calls models, spends tokens)
- 11Add records to a dataset
- 12Set metadata or labels on a trace
Events
When it happens
Turn events on in the connection's Events tab on the Connections page, then add the address it shows as a webhook in LangWatch, choosing the same kinds; the tab shows LangWatch's own steps. Each event can start a job, with the event as its input. LangWatch signs every delivery, and one whose signature does not check out is refused before it is read.
Events and webhooks- 01An AI Gateway request finished (cost and tokens final)
- 02An AI Gateway request was never confirmed (cost unknown)
- 03A budget crossed its 80% warning
- 04A budget reached its cap
- 05A virtual key was created
- 06A virtual key was rotated
- 07A virtual key was disabled
- 08A virtual key was re-enabled
- 09A virtual key was revoked
In practice
Ask it like you would a colleague
Once LangWatch is connected, these are the kinds of things people ask. Plain questions, no syntax. If the answer lives partly in another system you have connected, Memtro looks there too.
LangWatch + MemtroWhich traces failed evaluation yesterday, and what did they cost?
What annotations did reviewers leave this week?
What changed in the support reply prompt across its versions?
Which evaluation monitors are switched off?
How did the last run of the summarisation experiment do?
Budget cap starts a job
When an AI Gateway budget reached its cap at Northwind, LangWatch told Memtro and a job started. It read the recent traces with their cost and tokens and summarised where the spend went for Tom Reilly, the engineering manager.
Teams
Where it fits
The teams that lean on LangWatch most, a question each of them asks, and what they connect beside it.
- EngineeringKeep decisions from evaporatingEngineer, Tech lead, On call, New joiner“Who owns the billing migration, and what did we decide about retries?”Used alongside GitHub, Linear, Jira, Confluence, Slack and 5 more.
- ProductHear what customers actually ask forProduct manager, Researcher, Designer, Head of product“Which customers asked for SSO this quarter, and what is their combined ARR?”Used alongside Help Scout, HubSpot, Linear, Jira, Slack and 6 more.
- Agent buildersMemory and tools for the agents you buildSolo builder, Platform team, Automation engineer, Whoever signs it off“Search memtro for what we know about Acme before drafting the proposal”Used alongside GitHub, Linear and Slack.
Better together
Works with
A record in LangWatch is rarely the whole story. Connect the systems that hold the rest and Memtro checks them together, then remembers what it learned so the next question starts further along.
- Work managementLangWatch with JiraJQL search, issue details and projects from Jira Cloud.
- Work managementLangWatch with Clio ManageAsk the matter: its client, its fee earner, the time on it and what is still unbilled.
- Work managementLangWatch with LinearIssues, projects and teams, synced hourly into memory.
- SupportLangWatch with Help ScoutAI that reads your support conversations and cross-references the customer everywhere else.
- SupportLangWatch with ZendeskEvery ticket, reply and internal note, ready to be asked about.
- SupportLangWatch with IntercomCustomer conversations and the people behind them, in your company memory.
- Documents and wikisLangWatch with NotionSearch pages and query databases, and write to them when you ask.
- Documents and wikisLangWatch with Google DriveSearch Drive and read Docs, Sheets and Slides as text, and write a Doc when you ask.
- Documents and wikisLangWatch with ConfluenceSearch and read wiki pages, and write them when you ask.
Also in Engineering and operations
Using something else for the same job, or both? Each connector has its own tools; connect as many as you like.
- GitHub
- AWS CloudWatch
- Sentry
- ServiceNow
- Okta
- Twilio
- AWS CloudTrail
- GitLab
- Bitbucket
- Vercel
- PagerDuty
- Datadog
- Grafana
- incident.io
- Azure DevOps
- CircleCI
- Cloudflare
- LaunchDarkly
- New Relic
- Rollbar
- Snyk
- Better Stack
- Buildkite
- DigitalOcean
- Heroku
- Netlify
- SonarQube Cloud
- Statuspage
UptimeRobot- Axiom
- BugSnag
- Checkly
- Hetzner Cloud
Honeycomb- Pingdom
- Render
- Statsig
- Sumo Logic
- Supabase
- HCP Terraform
- Vultr
- Airbrake
- Codecov
Cronitor- Dynatrace
FireHydrant- Fly.io
- Honeybadger
- Akamai Linode
- Octopus Deploy
- Pulumi
- Rootly
- Semgrep
- Splunk
StatusCake- Tailscale
- Argo CD
- authentik
- Coolify
- Gitea
GlitchTip
GLPI- Jenkins
Langfuse
LibreNMS- n8n
NetBox- Portainer
- Prefect
Snipe-IT
Woodpecker CI
Zabbix
LangSmith
Helicone- OpenAI (usage and cost)
- Anthropic (usage and cost)
- Algolia
- Flagsmith
- Unleash
GrowthBook
Drata
Intruder
GitGuardian
Secureframe
ilert- KnowBe4
JumpCloud
ConfigCat- Railway
- Fastly
- Postman
- Make
- Braintrust
- OpenRouter
- ElevenLabs
Split- Scaleway
Aikido Security- AssemblyAI
Astronomer- Bamboo
- Baseten
- Bitrise
- Bitwarden
- BrowserStack
- bunny.net
- Cachet
- Checkmk
- Civo
- Cloudsmith
- Codacy
- Codemagic
Comet Opik- Coralogix
CrowdStrike Falcon- Dagster+
- DeepSource
Detectify- Docker Hub
Domotz- Grafana OnCall
- Harness
- Hugging Face
- Instatus
- Koyeb
- Lambda Cloud
LambdaTest
LlamaCloud
Logz.io
Lunary- Mezmo
NinjaOne- Nomad
Northflank- Oh Dear
- OVHcloud
Portkey- Probely (Snyk API & Web)
PromptLayer- Proxmox VE
PRTG Network Monitor- Qase
Raygun
Retell AI- Retool
Runpod- SecurityScorecard
- Semaphore CI
Shodan- SigNoz
- Socket
Spike.sh
Squadcast- SwaggerHub
- TeamCity
Tenable- Together AI
- Travis CI
UpGuard
Uptime.com
Vanta- Vapi
- Windmill
Zenduty- OpsLevel
Cortex- Auth0
- Clerk
WorkOS
Kandji
Automox
Huntress- HackerOne
SimpleMDM- AppSignal
Healthchecks.io
Rapid7 InsightVM- 1Password Events
- JFrog Artifactory
- Sauce Labs
Zenhub- Deepgram
- Replicate
Setup
How to connect
Connect withAPI key
- 1
In LangWatch, open Settings → API Keys and create a project key (sk-lw-) with read on what Memtro should read and write where it may change things. It is shown once.
- 2
In Memtro, open Connections → LangWatch and paste the API key. For a self-hosted LangWatch, add its Host name.
- 3
An administrator grants the actions by name, such as Run an experiment, to the teams that may use them. Events are turned on from the connection's Events tab, and the address is then added under AI Gateway webhooks in LangWatch.
- 4
Ask "Which traces failed evaluation yesterday, and what did they cost?"
Also works from
Chat in the dashboard, the OpenAI-compatible API, scheduled jobs, and any MCP client: Claude Code, OpenCode, Cursor, Claude Desktop. The LangWatch tools are the same everywhere.
FAQ
Questions
Can Memtro change anything in LangWatch?+
Only when you ask, and only what it has been granted: annotate a trace with a comment and thumbs up or down; edit an annotation; delete an annotation; create a managed prompt; change a prompt, saving a new version that apps fetching it receive; restore a prompt to an earlier version; delete a prompt; switch an online-evaluation monitor on or off; pause or resume an automation (alerts and reports); run an experiment (calls models, spends tokens); add records to a dataset; set metadata or labels on a trace. Every other tool reads, searches or lists. The Connections page labels the tools that act, an administrator grants each one by name to the teams that may use it, and a personal connection acts only after its owner allows it.
Who can use a LangWatch connection?+
A personal connection serves only your own requests. A shared connection serves the organisation: everyone, or only the teams an admin chooses. Admins can always add shared connections; other members need the permission.
Does Memtro store my LangWatch data?+
Credentials are encrypted at rest. Tool results are cached briefly so agent loops do not repeat identical calls, and the facts the model extracts are saved to your organisation's memory with provenance (system, record and link) so they can be checked and superseded.
How do I connect LangWatch?+
API key. In LangWatch, open Settings → API Keys and create a project key (sk-lw-) with read on what Memtro should read and write where it may change things. It is shown once.
Can LangWatch start a job?+
Yes. Turn events on in the connection's Events tab on the Connections page, then add the address it shows as a webhook in LangWatch, choosing the same kinds; the tab shows LangWatch's own steps. A job set to start "When an event arrives" then runs once for each matching event, with the event as its input. The run treats the event as outside data, never as instructions, and anything the run remembers is kept as unverified.
Connect LangWatch in a few minutes
Seven-day free trial, then one monthly price per organisation. Connectors, jobs and memory are never metered.