Explaining Eval Engineering | Galileo's Vikram Chatterji

https://is1-ssl.mzstatic.com/image/thumb/Podcasts211/v4/31/9a/aa/319aaaf4-a155-9f35-4383-74655a625ed5/mza_1227157986456528628.jpg/600x600bb.jpg

Chain of Thought

Conor Bronsdon

47 episodes

1 week ago

AI is reshaping infrastructure, strategy, and entire industries. Host Conor Bronsdon talks to the engineers, founders, and researchers building breakthrough AI systems about what it actually takes to ship AI in production, where the opportunities lie, and how leaders should think about the strategic bets ahead. Chain of Thought translates technical depth into actionable insights for builders and decision-makers. New episodes bi-weekly. Conor Bronsdon is an angel investor in AI and dev tools, Head of Technical Ecosystem at Modular, and previously led growth at AI startups Galileo and LinearB.

Technology

RSS

All content for Chain of Thought is the property of Conor Bronsdon and is served directly from their servers with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.

Technology

https://d3t3ozftmdmh3i.cloudfront.net/staging/podcast_uploaded_nologo/42310277/42310277-1763428809518-e309cc0a3f9d.jpg

Explaining Eval Engineering | Galileo's Vikram Chatterji

Chain of Thought

37 minutes 14 seconds

3 weeks ago

Explaining Eval Engineering | Galileo's Vikram Chatterji

You've heard of evaluations—but eval engineering is the difference between AI that ships and AI that's stuck in prototype.

Most teams still treat evals like unit tests: write them once, check a box, move on. But when you're deploying agents that make real decisions, touch real customers, and cost real money, those one-time tests don't cut it. The companies actually shipping production AI at scale have figured out something different—they've turned evaluations into infrastructure, into IP, into the layer where domain expertise becomes executable governance.

Vikram Chatterji, CEO and Co-founder of Galileo, returns to Chain of Thought to break down eval engineering: what it is, why it's becoming a dedicated discipline, and what it takes to actually make it work. Vikram shares why generic evals are plateauing, how continuous learning loops drive accuracy, and why he predicts "eval engineer" will become as common a role as "prompt engineer" once was.

In this conversation, Conor and Vikram explore:

Why treating evals as infrastructure—not checkboxes—separates production AI from prototypes
The plateau problem: why generic LLM-as-a-judge metrics can't break 90% accuracy
How continuous human feedback loops improve eval precision over time
The emerging "eval engineer" role and what the job actually looks like
Why 60-70% of AI engineers' time is already spent on evals
What multi-agent systems mean for the future of evaluation
Vikram's framework for baking trust AND control into agentic applications

Plus: Conor shares news about his move to Modular and what it means for Chain of Thought going forward.

Chapters:00:00 – Introduction: Why Evals Are Becoming IP01:37 – What Is Eval Engineering?04:24 – The Eval Engineering Course for Developers05:24 – Generic Evals Are Plateauing08:21 – Continuous Learning and Human Feedback11:01 – Human Feedback Loops and Eval Calibration13:37 – The Emerging Eval Engineer Role16:15 – What Production AI Teams Actually Spend Time On18:52 – Customer Impact and Lessons Learned24:28 – Multi-Agent Systems and the Future of Evals30:27 – MCP, A2A Protocols, and Agent Authentication33:23 – The Eval Engineer Role: Product-Minded + Technical34:53 – Final Thoughts: Trust, Control, and What's Next

Connect with Conor Bronsdon:Substack – https://conorbronsdon.substack.com/LinkedIn – https://www.linkedin.com/in/conorbronsdon/X (Twitter) – https://x.com/ConorBronsdon

Learn more about Eval Engineering:⁠https://galileo.ai/evalengineering⁠

Connect with Vikram Chatterji:LinkedIn – ⁠https://www.linkedin.com/in/vikram-chatterji/⁠