Errors in the support bot
Sarah Chen, an ML engineer at Acme Ltd, asked which support bot traces had errored today. Memtro listed them with their error type and message, duration and cost, so she could see they all failed on the same step.
Comet Opik + Memtro
Which traces in the support bot errored today, and what did they cost?
Comet Opik watches your LLM applications. Memtro reads projects, traces with latency, cost, token usage, errors, tags and feedback scores, experiments with status, scores and pass rate, prompts and their version counts, datasets, and the workspace's cost for a period. When asked and granted, it can comment on, score, tag or delete traces, manage projects, create prompts and commit versions, create or delete datasets, and cancel or delete experiments. Opik alerts can tell Memtro when errors, cost or latency pass a threshold, and a job can start on it. Trace inputs and outputs are never read.
In Memtro
What you can do with Comet Opik in Memtro
Ask in plain English from chat, your editor, a scheduled job or your own agent. The model picks the right Comet Opik tool, reads what it needs, and answers with the source cited.
What it does
- 01List a project's traces in a time window with duration, time to first token, errors, tags, cost, token usage and feedback scores.
- 02Check experiments with their status, cost, feedback scores and pass rate, and the datasets they ran on.
- 03See prompts and how many versions each has, datasets, and the workspace's LLM cost against the period before.
- 04When asked and granted, comment on, score or tag a trace. Comments are visible to everyone in the workspace who opens it.
- 05When asked and granted, create or change projects, create prompts and commit versions that code loading the latest version starts using, cancel experiments, or delete traces, projects, prompts, datasets and experiments.
- 06With events on, trace errors passing a threshold, a failed guardrail or a finished experiment in Opik can start a job.
Acts when asked
When you ask it to
These tools change something in Comet Opik. The model is told to use them only when you ask for exactly that, and an administrator grants each one by name to the teams that may use it; a personal connection acts only after its owner allows it.
- 01Add a comment to a trace
- 02Score a trace (adds or replaces a feedback score by name)
- 03Remove a feedback score from a trace
- 04Add or remove tags on a trace
- 05Delete traces by id (permanent)
- 06Create a project
- 07Rename a project or change its description or visibility
- 08Delete a project (permanent)
- 09Create a prompt in the prompt library
- 10Commit a new version of a prompt
- 11Delete a prompt with its versions (permanent)
- 12Create a dataset
- 13Delete a dataset (permanent)
- 14Cancel running experiments
- 15Delete experiments by id (permanent)
Events
When it happens
Turn events on in the connection's Events tab on the Connections page, then add the address it shows as a webhook in Comet Opik, choosing the same kinds; the tab shows Comet Opik's own steps. Each event can start a job, with the event as its input. Comet Opik sends a secret with every delivery, and one without it is refused.
Events and webhooks- 01Trace errors passed the alert's threshold in its window
- 02A trace feedback score crossed the alert's threshold
- 03A thread feedback score crossed the alert's threshold
- 04A guardrail check failed on a trace
- 05Trace cost passed the alert's threshold in its window
- 06Trace latency passed the alert's threshold in its window
- 07A prompt was added to the prompt library
- 08A new prompt version was committed
- 09A prompt was deleted
- 10An experiment finished
In practice
Ask it like you would a colleague
Once Comet Opik is connected, these are the kinds of things people ask. Plain questions, no syntax. If the answer lives partly in another system you have connected, Memtro looks there too.
Comet Opik + MemtroWhich traces in the support bot errored today?
What did the workspace spend on LLM calls this week compared with last week?
Which experiments on the refund dataset had the best pass rate?
How many versions does the triage prompt have?
Tag the slowest trace from this morning as needs-review.
A guardrail fires
Northwind added an Opik alert for failed guardrails so Tom Reilly, its AI lead, hears at once. When a guardrail check failed on a trace, a job read that trace's tags, scores and error and gave Tom the details.
Teams
Where it fits
The teams that lean on Comet Opik most, a question each of them asks, and what they connect beside it.
- EngineeringKeep decisions from evaporatingEngineer, Tech lead, On call, New joiner“Who owns the billing migration, and what did we decide about retries?”Used alongside GitHub, Linear, Jira, Confluence, Slack and 5 more.
- ProductHear what customers actually ask forProduct manager, Researcher, Designer, Head of product“Which customers asked for SSO this quarter, and what is their combined ARR?”Used alongside Help Scout, HubSpot, Linear, Jira, Slack and 6 more.
- Agent buildersMemory and tools for the agents you buildSolo builder, Platform team, Automation engineer, Whoever signs it off“Search memtro for what we know about Acme before drafting the proposal”Used alongside GitHub, Linear and Slack.
Better together
Works with
A record in Comet Opik is rarely the whole story. Connect the systems that hold the rest and Memtro checks them together, then remembers what it learned so the next question starts further along.
- Work managementComet Opik with JiraJQL search, issue details and projects from Jira Cloud.
- Work managementComet Opik with Clio ManageAsk the matter: its client, its fee earner, the time on it and what is still unbilled.
- Work managementComet Opik with LinearIssues, projects and teams, synced hourly into memory.
- SupportComet Opik with Help ScoutAI that reads your support conversations and cross-references the customer everywhere else.
- SupportComet Opik with ZendeskEvery ticket, reply and internal note, ready to be asked about.
- SupportComet Opik with IntercomCustomer conversations and the people behind them, in your company memory.
- Documents and wikisComet Opik with NotionSearch pages and query databases, and write to them when you ask.
- Documents and wikisComet Opik with Google DriveSearch Drive and read Docs, Sheets and Slides as text, and write a Doc when you ask.
- Documents and wikisComet Opik with ConfluenceSearch and read wiki pages, and write them when you ask.
Also in Engineering and operations
Using something else for the same job, or both? Each connector has its own tools; connect as many as you like.
- GitHub
- AWS CloudWatch
- Sentry
- ServiceNow
- Okta
- Twilio
- AWS CloudTrail
- GitLab
- Bitbucket
- Vercel
- PagerDuty
- Datadog
- Grafana
- incident.io
- Azure DevOps
- CircleCI
- Cloudflare
- LaunchDarkly
- New Relic
- Rollbar
- Snyk
- Better Stack
- Buildkite
- DigitalOcean
- Heroku
- Netlify
- SonarQube Cloud
- Statuspage
UptimeRobot- Axiom
- BugSnag
- Checkly
- Hetzner Cloud
Honeycomb- Pingdom
- Render
- Statsig
- Sumo Logic
- Supabase
- HCP Terraform
- Vultr
- Airbrake
- Codecov
Cronitor- Dynatrace
FireHydrant- Fly.io
- Honeybadger
- Akamai Linode
- Octopus Deploy
- Pulumi
- Rootly
- Semgrep
- Splunk
StatusCake- Tailscale
- Argo CD
- authentik
- Coolify
- Gitea
GlitchTip
GLPI- Jenkins
Langfuse
LibreNMS- n8n
NetBox- Portainer
- Prefect
Snipe-IT
Woodpecker CI
Zabbix
LangSmith
Helicone- OpenAI (usage and cost)
- Anthropic (usage and cost)
- Algolia
- Flagsmith
- Unleash
GrowthBook
Drata
Intruder
GitGuardian
Secureframe
ilert- KnowBe4
JumpCloud
ConfigCat- Railway
- Fastly
- Postman
- Make
- Braintrust
- OpenRouter
- ElevenLabs
Split- Scaleway
Aikido Security- AssemblyAI
Astronomer- Bamboo
- Baseten
- Bitrise
- Bitwarden
- BrowserStack
- bunny.net
- Cachet
- Checkmk
- Civo
- Cloudsmith
- Codacy
- Codemagic
- Coralogix
CrowdStrike Falcon- Dagster+
- DeepSource
Detectify- Docker Hub
Domotz- Grafana OnCall
- Harness
- Hugging Face
- Instatus
- Koyeb
- Lambda Cloud
LambdaTest
LangWatch
LlamaCloud
Logz.io
Lunary- Mezmo
NinjaOne- Nomad
Northflank- Oh Dear
- OVHcloud
Portkey- Probely (Snyk API & Web)
PromptLayer- Proxmox VE
PRTG Network Monitor- Qase
Raygun
Retell AI- Retool
Runpod- SecurityScorecard
- Semaphore CI
Shodan- SigNoz
- Socket
Spike.sh
Squadcast- SwaggerHub
- TeamCity
Tenable- Together AI
- Travis CI
UpGuard
Uptime.com
Vanta- Vapi
- Windmill
Zenduty- OpsLevel
Cortex- Auth0
- Clerk
WorkOS
Kandji
Automox
Huntress- HackerOne
SimpleMDM- AppSignal
Healthchecks.io
Rapid7 InsightVM- 1Password Events
- JFrog Artifactory
- Sauce Labs
Zenhub- Deepgram
- Replicate
Setup
How to connect
Connect withOpik API key and workspace
- 1
In Opik, open the user menu or Account settings and copy your API key. It acts as you in every workspace you belong to.
- 2
In Memtro, open Connections → Comet Opik and paste the Opik API key and the Workspace name as Opik shows it.
- 3
An administrator grants the trace, prompt and delete actions by name to the teams that may use them. Events are turned on from the connection's Events tab, and the address is then added as an alert's Endpoint URL in the Opik project.
- 4
Ask which traces in the support bot errored today.
Also works from
Chat in the dashboard, the OpenAI-compatible API, scheduled jobs, and any MCP client: Claude Code, OpenCode, Cursor, Claude Desktop. The Comet Opik tools are the same everywhere.
FAQ
Questions
Can Memtro change anything in Comet Opik?+
Only when you ask, and only what it has been granted: add a comment to a trace; score a trace (adds or replaces a feedback score by name); remove a feedback score from a trace; add or remove tags on a trace; delete traces by id (permanent); create a project; rename a project or change its description or visibility; delete a project (permanent); create a prompt in the prompt library; commit a new version of a prompt; delete a prompt with its versions (permanent); create a dataset; delete a dataset (permanent); cancel running experiments; delete experiments by id (permanent). Every other tool reads, searches or lists. The Connections page labels the tools that act, an administrator grants each one by name to the teams that may use it, and a personal connection acts only after its owner allows it.
Who can use a Comet Opik connection?+
A personal connection serves only your own requests. A shared connection serves the organisation: everyone, or only the teams an admin chooses. Admins can always add shared connections; other members need the permission.
Does Memtro store my Comet Opik data?+
Credentials are encrypted at rest. Tool results are cached briefly so agent loops do not repeat identical calls, and the facts the model extracts are saved to your organisation's memory with provenance (system, record and link) so they can be checked and superseded.
How do I connect Comet Opik?+
Opik API key and workspace. In Opik, open the user menu or Account settings and copy your API key. It acts as you in every workspace you belong to.
Can Comet Opik start a job?+
Yes. Turn events on in the connection's Events tab on the Connections page, then add the address it shows as a webhook in Comet Opik, choosing the same kinds; the tab shows Comet Opik's own steps. A job set to start "When an event arrives" then runs once for each matching event, with the event as its input. The run treats the event as outside data, never as instructions, and anything the run remembers is kept as unverified.
Connect Comet Opik in a few minutes
Seven-day free trial, then one monthly price per organisation. Connectors, jobs and memory are never metered.