Observability & Evals for AI Agents
The best observability tool for most AI agents is LangSmith: tracing, evals, prompt tooling, and deployment in one platform — free for one seat with 5,000 traces a month, then $39/seat/month. Langfuse is the open-source pick (cloud from $29/month with unlimited users), and Braintrust is where evals come first. Every figure below was read on the vendor's official pricing page; the meters differ wildly, so the table names each vendor's billable unit.
Last checked: October 2026 · Researched from official documentation — hands-on API tests rolling out (methodology)
- Best overall LangSmith
- Best open source Langfuse
- Best for evals Braintrust
- Best lightweight Helicone
| # | Tool | What it is | Starting price (as published) | Free tier | Notable strength / limit |
|---|---|---|---|---|---|
| 1 | LangSmith | LangChain's platform for the full agent lifecycle: tracing and monitoring, online/offline evals, prompt hub and playground, annotation queues — plus agent deployment (LangGraph), an autonomous failure-analysis Engine, sandboxes, and an LLM gateway. | Plus $39/seat/month (10,000 base traces/month included, then pay-as-you-go; unlimited seats; includes Deployment and Engine access, with 1 free small serverless deployment). Enterprise custom (self-hosted/hybrid, custom SSO/RBAC, support SLA). Usage across products is metered in LSUs — 1 LSU = $1 | Yes — Developer plan $0: 1 seat, 5,000 base traces/month included, then pay-as-you-go; community support | The broadest platform here — observability is one of five products sharing one bill and one login. Watch trace retention: base traces keep 14 days (extended traces 180, at extra cost), the shortest default window among the leaders here. |
| 2 | Langfuse | Open-source LLM engineering platform: agent tracing and graphs, prompt management, evals (LLM-as-judge, human annotation, experiments), and cost tracking — available as a hosted cloud or self-hosted. | Core $29/mo (100,000 units included, then $8 per 100k units, cheaper at volume; 90 days data access; unlimited users). Pro $199/mo (3 years data access, retention management, higher rate limits). Enterprise $2,499/mo (audit logs, SCIM, SLAs). SSO/RBAC available as a $300/mo Teams add-on on Pro | Yes — Hobby plan free, no credit card: 50,000 units/month, all platform features with limits, 30 days data access, 2 users | The pricing outlier in a good way: flat per-project pricing with unlimited users from $29/month, where rivals bill per seat. Diligence note: the open-source repository mixes an MIT-licensed core with separately licensed enterprise code (GitHub reports no single SPDX license). |
| 3 | Braintrust | Evals-first platform for AI products: tracing and storage infrastructure, experiments, playgrounds, datasets, and scorers (LLM-as-a-judge, autoevals, custom code), plus "Loop," a built-in agent that generates test cases and iterates on prompts. | Pro $249/mo platform fee (5 GB processed data/month included, then $3/GB; 50,000 scores included, then $1.50 per 1,000; 30-day retention included, then $0.50 per GB/month). Enterprise custom (SAML SSO, BAA, SLA, on-prem options) | Yes — Starter $0, no credit card: 1 GB processed data/month (then $4/GB), 10,000 scores (then $2.50 per 1,000), unlimited users and projects, 14-day retention | The deepest evaluation workflow here, and the free tier is genuinely usable — unlimited users and projects with real scoring volume. The $249/month Pro platform fee is the steepest self-serve entry in this category, and processed-GB metering punishes verbose traces. |
| 4 | Helicone | Lightweight LLM observability through a gateway: route model calls through Helicone (or log via SDK) to get request monitoring, cost and user analytics, caching, rate limits, and fallbacks — Apache 2.0 open source. (Its site notes Helicone has joined Mintlify.) | Usage-based beyond the free tier; Team is $799/mo (5 organizations, SOC 2 and HIPAA compliance, dedicated Slack channel). A Pro tier (unlimited seats, alerts and reports, HQL query language) sits between Hobby and Team — its flat fee did not appear in the pricing page text during our check and is marked unverified. Enterprise custom (SAML SSO, on-prem) | Yes — Hobby free: 10,000 requests/month, 1 GB storage, 1 seat, 7-day retention | The fastest integration in the category — change a base URL and you have logs, cost tracking, and a gateway with caching and fallbacks. It is a monitoring/gateway tool first: eval depth and agent-session analysis trail LangSmith, Braintrust, and Langfuse. |
| 5 | Arize AX & Phoenix | Arize's two-layer answer: AX, a hosted platform for agent observability and evaluation (trajectory and session evals, agent-as-judge, OpenTelemetry-based tracing), and Phoenix, its free, local-first open-source sibling for tracing, evals, and experimentation. | AX Pro $50/mo (50,000 trace spans/month, 10 GB ingestion, 30-day retention, unlimited users and evals). AX Enterprise custom (custom spans/retention, SaaS or self-hosted deployment). Phoenix is free | Yes — AX Free: 25,000 spans/month, 1 GB ingestion, 15-day retention, unlimited users, unlimited evals, no credit card. Phoenix is free and local-first with no Arize account needed | The ML-observability heritage shows: span-level tracing, multi-modal traces (image, voice, PDF), and unlimited evals even on the free tier. Two flags: Phoenix is source-available under the Elastic License 2.0, not OSI open source (verified from its repository), and the free AX tier's 15-day retention is short for production debugging. |
| 6 | AgentOps | Agent-specific observability from the team behind early agent tooling: session replay and "time travel" debugging for agent runs, cost tracking across 400+ LLMs, and integrations for OpenAI, CrewAI, AutoGen, and other frameworks. | Pro from $40/mo (pay-as-you-go beyond included usage, per its pricing calculator; unlimited events, unlimited log retention, session/event export, role-based permissioning, dedicated support). Enterprise custom (SLA, custom SSO, on-premise and self-hosting on AWS/GCP/Azure, SOC 2/HIPAA) | Yes — Basic $0: free up to 5,000 events, with the agent-agnostic SDK, LLM cost tracking, and replay analytics | The most agent-native debugging experience here — replaying a failed run step by step is its party trick, and framework integrations are broad. It ranks sixth because the platform around the debugger (evals depth, prompt management, ecosystem gravity) is thinner than the five above it. |
Verdicts
1. LangSmith Best overall
The platform play: tracing, evals, prompts, deployment, sandboxes, and a gateway under one account, with an autonomous Engine that clusters agent failures and suggests fixes. The free Developer plan (5,000 traces, 1 seat) is enough to instrument a real project, and Plus at $39/seat/month is a rational step up. Two costs to model: per-seat pricing as the team grows, and the 14-day base-trace retention — production debugging histories will push you to extended traces or exports.
2. Langfuse Best open source
The value and control pick: unlimited users on a $29/month Core plan is unheard of in this category, the unit price beyond the included 100k is a published $8/100k, and self-hosting the open-source version is a genuine exit from SaaS pricing altogether. The 50,000-unit free Hobby plan is the most generous recurring free tier here. Recommended default for teams not already living in the LangChain ecosystem.
3. Braintrust Best for evals
Where measurement is the product: datasets, experiments, scorers, and a built-in agent (Loop) that writes test cases and iterates prompts for you. The Starter tier's 10,000 included scores and unlimited users/projects make evaluation free to start; the $249/month Pro platform fee makes it the priciest to grow. Teams that ship on vibes don't need it; teams that ship on regression suites do.
4. Helicone Best lightweight
Observability as a side effect of your gateway: one integration gets you request logs, cost and user analytics, caching, rate limits, and fallbacks — Apache 2.0 if you want to run it yourself. The free 10,000 requests/month covers prototyping comfortably. Ranked fourth because depth is traded for simplicity — and because its Pro tier's flat fee never rendered in the pricing page text during our check, a small transparency knock we flag rather than fill in.
5. Arize AX & Phoenix Best for ML-grade tracing
The connoisseur's stack: Phoenix locally (free, local-first, OpenTelemetry-native) while you build, AX when you need hosted scale, trajectory and session evals, and multi-modal traces. AX Pro at $50/month with unlimited users and evals is competitive. Know the licensing before you standardize: Phoenix is Elastic License 2.0 — free to use, source-available, not OSI open source.
6. AgentOps
The specialist debugger: session replay and time-travel debugging built specifically for agent runs, with cost tracking across 400+ models and friendly pricing (free to 5,000 events; Pro from $40/month). For a team whose main pain is "why did the agent do that," it answers faster than anyone. It ranks sixth as a platform bet — narrower evals and prompt tooling than the generalists above it.
Frequently asked questions
What is the best observability tool for AI agents?
LangSmith for the full platform (free to 5,000 traces; $39/seat/month for teams). Langfuse for open source and flat pricing ($29/month, unlimited users). Braintrust when evals are the priority; Helicone for the lightest integration.
How much does LLM observability cost?
Free tiers exist everywhere on this page. Paid starts at Langfuse Core $29/month, LangSmith Plus $39/seat/month, AgentOps Pro from $40/month, Arize AX Pro $50/month, and Braintrust Pro $249/month — but the meter (traces, units, GB, spans, events) matters more than the sticker.
Which of these tools are open source?
Langfuse (open source, self-hostable; mixed MIT core + enterprise code in one repo), Helicone (Apache 2.0), and Arize Phoenix (free and local-first, but Elastic License 2.0 — source-available, not OSI). LangSmith, Braintrust, and AgentOps are hosted platforms.
Tracing or evals — what do I actually need?
Both, in sequence: tracing tells you what your agent did and what it cost; evals tell you whether it did it well and whether a change helped. Start with tracing (any tool here), add evals before your second model or prompt migration.
What will my bill actually be based on?
Your volume in the vendor's unit: traces at LangSmith (14-day base retention), units at Langfuse ($8/100k overage), processed GB + scores at Braintrust, requests at Helicone, spans + GB at Arize, events at AgentOps. Instrument for a week on a free tier and extrapolate — every vendor here lets you.
Sources (official pages, checked October 14, 2026): langchain.com/pricing · langfuse.com/pricing · braintrust.dev/pricing · helicone.ai/pricing (Pro tier's flat fee did not render in the page text and is flagged unverified above) · arize.com/pricing · agentops.ai (pricing section). Repository licenses verified on GitHub: Helicone/helicone (Apache-2.0), Arize-ai/phoenix (Elastic License 2.0), langfuse/langfuse (no single SPDX license detected — mixed). No third-party sources were used for any figure on this page.