Relari (YC W24)Testing, evaluation, and synthetic data generation platform for AI agents.

4.3 (6)

Reviewed by Daniel Nikulshyn·Updated July 2026

Overview

Relari is a developer platform focused on improving the reliability of AI agents through systematic testing and evaluation. It helps teams generate synthetic datasets, run automated evaluations, and benchmark agent performance across realistic scenarios before shipping to production. Backed by Y Combinator (W24), Relari targets engineering teams building complex LLM applications and multi-step agents where traditional QA falls short. Its tooling aims to bring software-engineering rigor—unit tests, regression checks, and measurable metrics—to non-deterministic AI systems. The platform supports custom evaluators, scenario simulation, and continuous monitoring, making it useful for both pre-launch validation and ongoing quality assurance of production agents.

Key features

Synthetic dataset generation
Automated agent evaluation pipelines
Scenario and conversation simulation
Customizable evaluation metrics
Regression testing for LLM apps
Performance benchmarking and reporting

Pricing

Model: Free
Category: Observability
Rating: 4.3 / 5 (6)

Use cases

Testing AI agents

Reliable and testable agents evaluation for AI agents.

Pros & Cons

Pros

Purpose-built for evaluating multi-step AI agents
Generates synthetic test data at scale
Supports custom metrics and evaluators
Backed by Y Combinator with active development

Cons

Primarily aimed at technical teams, not non-developers
Newer platform with an evolving feature set
May require integration work to fit existing stacks

Reviews

4.3

Average from 6 ratings.

Fatima Zahra

Apr 4, 2026

Skeptical, then convinced

I went in skeptical — most tools in this space overpromise. It actually delivers on customizable evaluation metrics, and purpose-built for evaluating multi-step AI agents caught me off guard. still, I'd recommend giving it a real trial.

Robert Ainsworth

Mar 17, 2026

Solid for our team

We rolled this out across the team last quarter and supports custom metrics and evaluators. Customizable evaluation metrics fits neatly into how we already work, and customizable evaluation metrics removed a step we used to do by hand. Primarily aimed at technical teams, not non-developers, which is the main caveat, but it has held up under daily use.

Devin Walker

Feb 6, 2026

Skeptical, then convinced

I went in skeptical — most tools in this space overpromise. It actually delivers on performance benchmarking and reporting, and supports custom metrics and evaluators caught me off guard. Primarily aimed at technical teams, not non-developers is why this isn't a perfect score, still, I'd recommend giving it a real trial.

Carlos Mendoza

Jul 20, 2025

Compared a few options

Evaluated this against two competitors. Where it wins: scenario and conversation simulation and purpose-built for evaluating multi-step AI agents. Where it lags: may require integration work to fit existing stacks. On balance the feature set — especially scenario and conversation simulation — justifies the 5 stars for our use case.

Yuki Mori

Jul 19, 2025

Use it every day

Honestly didn't expect to like it this much. Performance benchmarking and reporting is exactly what I needed, and purpose-built for evaluating multi-step AI agents. I do wish may require integration work to fit existing stacks, but I reach for it almost every day now and it just clicks.

Leila Hassan

Jul 14, 2025

Skeptical, then convinced

I went in skeptical — most tools in this space overpromise. It actually delivers on regression testing for LLM apps, and supports custom metrics and evaluators caught me off guard. May require integration work to fit existing stacks is why this isn't a perfect score, still, I'd recommend giving it a real trial.

Q&A

No questions yet — be the first to ask.

Ask a question

Observability alternatives

KeywordsAI

Observability

Unified developer platform for building, monitoring, and scaling LLM applications.

5.0 (6)

Free

Guardian

Observability

Security and governance platform for autonomous AI agents and intelligent systems.

5.0 (5)

Free

Maxim AI

Observability

End-to-end platform for evaluating, monitoring, and improving AI agents

4.8 (6)

Free

Weave

Observability

A no-code AI workflow builder that enables businesses to automate operations by integrating multiple large language models (LLMs) and connecting prompts seam...

4.8 (5)

Free