Livemarks

Good match?

Made this video to explain evals

Good match?

evals evals evals evals

Good match?

I often hear that evals are the most confusing part of creating LLM AI products. It's a shame b/c IMO, domain-specific evals are the most important part of an AI product! I've written a detailed blog post with real examples on how to do this (1/3) https://t.co/nCUPWGcn33 https://hamel.dev/blog/posts/evals/

Good match?

I expanded the evals FAQ by 10+ questions, and improved many of the old answers. Exact section and question titles below 1. Getting Started & Fundamentals • Q: What are LLM Evals? • Q: What is a trace? • Q: What’s a minimum viable evaluation setup? • Q: How much of my development budget should I allocate to evals? • Q: Will today’s evaluation methods still be relevant in 5–10 years given how fast AI is changing? 2. Error Analysis & Data Collection • Q: Why is "error analysis" so important in LLM evals, and how is it performed? • Q: How do I surface problematic traces for review beyond user feedback? • Q: How often should I re-run error analysis on my production system? • Q: What is the best approach for generating synthetic data? • Q: Are there scenarios where synthetic data may not be reliable? • Q: How do I approach evaluation when my system handles diverse user queries? • Q: How can I efficiently sample production traces for review? 3. Evaluation Design & Methodology • Q: Why do you recommend binary (pass/fail) evaluations instead of 1–5 ratings (Likert scales)? • Q: Should I practice eval-driven development? • Q: Should I build automated evaluators for every failure mode I find? • Q: Should I use "ready-to-use" evaluation metrics? • Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs? • Q: Can I use the same model for both the main task and evaluation? 4. Human Annotation & Process • Q: How many people should annotate my LLM outputs? • Q: Should product managers and engineers collaborate on error analysis? How? • Q: Should I outsource annotation & labeling to a third party? • Q: What parts of evals can be automated with LLMs? • Q: Should I stop writing prompts manually in favor of automated tools? 5. Tools & Infrastructure • Q: Should I build a custom annotation tool or use something off-the-shelf? • Q: What makes a good custom interface for reviewing LLM outputs? • Q: What gaps in eval tooling should I be prepared to fill myself? • Q: Seriously Hamel. Stop the bullshit. What’s your favorite eval vendor? 6. Production & Deployment • Q: How are evaluations used differently in CI/CD vs. monitoring production? • Q: What’s the difference between guardrails & evaluators? • Q: Can my evaluators also be used to automatically fix or correct outputs in production? • Q: How much time should I spend on model selection? 7. Domain-Specific Applications • Q: Is RAG dead? • Q: How should I approach evaluating my RAG system? • Q: How do I choose the right chunk size for my document processing tasks? • Q: How do I debug multi-turn conversation traces? • Q: How do I evaluate sessions with human handoffs? • Q: How do I evaluate complex multi-step workflows? • Q: How do I evaluate agentic workflows? Link: https://t.co/wi6vwJwokA https://hamel.dev/blog/posts/evals-faq/

Good match?

seriously this tidal wave of testimonials is insane ? https://t.co/cfebYMlr4x https://testimonial.to/ai-evals-course/all

Good match?

Great post on evals from @julianeagu who spent tons of time in the trenches working with evals on applied AI products at scale (GitHub CoPilot)

Good match?

PSA the recent evals debate on the timeline was born out of this war of "A/B testing vs Evals" between two vendors (read the post for details) You need both online and offline tests, they have different tradeoffs etc. One doesn't eat the other. - Offline metrics are proxies for behaviors and outcomes, and much faster to iterate on. - Online metrics and A/B testing are necessary to keep yourself in check and make sure your offline metrics are in fact, good proxies (also you can only measure certain things with online tests). I feel like we need to bring data science back into AI Eng because its absence is really showing in the discourse. @swyx has organized a forum for both parties to discuss on @latentspacepod next week, so I don't have to mute the word evals 😅

Good match?

Evals are really a Jedi mind trick to get you to try lots of experiments quickly and measure the results Its amazing when people realize this

Good match?

Should you engage in "Evals Driven Development"? Probably Not! I bet you thought I would tell everyone to evals max 😆. You have to be reasonable here.

Good match?

The Evals FAQ gets updated frequently - ones we just added: Q: How do we evaluate a model’s ability to express uncertainty or "know what it doesn’t know"? Q: How should I version and manage prompts? Q: How do I make the case for investing in evals to my team?

RazRazcle

@RazRazcle

Good match?

Building great AI products isn't rocket science but you need to get the fundamentals right. In the High Agency pod this week, @swyx and I interviewed @HamelHusain on how to build great AI evals. If you want to improve your AI products I strongly recommend giving it a listen.

Good match?

To what extent can you automate or delegate evals? Is there a way to make it "not your problem?" 😅 Part 1 of 1: You should absolutely automate parts of it as long as a human is in the loop. Many people are a bit too aggressive here, so you have to be careful, some guidelines below:

Good match?

Re: CaseText & eval driven development (EDD) for AI. EDD was always front and center pre-generative AI. Ex: nobody cares about your fraud/churn/forecasting model if it’s not accurate. This is missing with most LLM products. But the best practices and playbooks are there in classic ML! It’s a bit amusing that EDD is an epiphany (but glad there is attention being brought to it) Here are some posts that may be helpful 1. https://t.co/nCUPWGcn33 2. https://t.co/iGQXHaMtqm https://hamel.dev/blog/posts/evals/ https://eugeneyan.com/writing/evals/

Good match?

I started doing office hours on LLM evals and met with 8+ founders in the last 3 weeks. Common questions: - Which components of our app do we start evaluating (RAG,tool calls, etc)? - What metrics should I use? - Where should I spend my time? All have the same solution. LOOK AT THE DATA. What does this mean though? It means look at your logs/traces - start with 30 or so. Start categorizing the errors and issues you see. Keep looking at logs and traces until you feel like you aren't learning anything new. In the end, you will know where your biggest issues are. You prioritize those! You will also get a sense of what is most important to measure (and how). That's it. Look at data, build evals and tests prioritized by patterns in the data. If you don't have data, generate synthetic inputs/interactions into your LLM application so you can generate data. I didn't make this technique up fwiw. These are fundamentals of building machine learning systems and is often referred to as "Error Analysis". It is a fancy word for looking at data, categorizing errors, and then doing data analysis on those errors to understand what to prioritize and work on. I've documented some of the office hours, and you can see in all cases the solution was performing error analysis. Here are links to those: https://t.co/Z5CSGznzgP https://hamel.dev/notes/llm/officehours/

Good match?

This nearly 100% of companies I’ve talked to 🤣

Good match?

If your llm evals consist of off the shelf metrics pip install evals from evals import hallucination, conciseness, tone, sentiment You are wasting your time.

Good match?

I'm going to be showing how to do error analysis on LLM traces today Error analysis is the foundation of evals, b/c it clarifies: - Where to focus your efforts - What / how to measure failures - How to fix almost any problem tbh Today @ 4:30 PM PST https://t.co/CQH0fz1LoD https://lu.ma/ndy6buf1?utm_source=hh

Good match?

re: this diagram from @swyx. I think its an exciting time to "shift right" where possible. Things like FastHTML + copilot are allowing people with skills on the left to shift right easier than ever before (e.g. Python for everything) Previously I felt like it was too much surface area for mortals to span the entire width of the diagram, but it seems way more tractable now. I suspect there will be tools for people to "shift left" too (but I have far less insight into who is building those)

Good match?

📢New: Part 2 (of 3) What We Learned from a Year of Building with LLMs https://t.co/RE0OCdSCCy If you liked Part 1, Part 2 is a banger. We answer the following: Some of my favorite takes: AI Engineering Is NOT All You Need Look At Your Data We found that most people are not looking at their data as frequently as they should. You have to do this if you are serious about improving your AI Evals Are About Iterating Quickly Some people think that Evals are an academic exercise. But it is not! Evals dramatically increase the speed at which you can ship. You can read the full article here: https://t.co/RE0OCdSCCy You can follow the authors of this paper to hear additional commentary and hear about upcoming Part 3: @eugeneyan @BEBischof @charles_irl @sh_reya @jxnlco @HamelHusain https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-ii/

Good match?

I used the design plugin + Opus 4.5 to upgrade my business page https://t.co/b6C5EA4QtU Absolutely blown away. Before vs. After https://parlance-labs.com/

Good match?

# The second era of AI engineering > "The single biggest predictor of how rapidly a team makes progress building an AI agent lay in their ability to drive a disciplined process for evals (measuring the system’s performance) and error analysis (identifying the causes of errors)." The first era of AI engineering was justifiably characterized by gluing together tools and APIs. A significant proportion of products that achieved commercial success in the 1st era were coding agents, which benefitted from tremendous rigor & evals baked into post-training process. OTOH, Many people got burned by evals in this era because they demanded that evals should be "just another one of these tools that we plug in". This did not go well. In the second era, I believe we are going to see a resurgence of a persona like the data scientist [AI Scientist?] who is adept at looking through data to generate hypothesis, craft custom metrics, and debug stochastic systems. This will become increasingly valuable in many domains where we do not have the benefit of domain-specific post-training or dogfooding by foundation model labs (like is often the case with coding agents). It's exciting to see Andrew Ng independently arrive at this conclusion and champion it. Really looking forward to seeing more machine learning engineers and data scientists realize how valuable they are in applied AI. For anyone that wants to learn more about what this looks like IRL, I'll put a link to a YT video in the reply.

Good match?

Feels good to be mentioned on HN for engineers learning AI 🥰 Helping others is a big reason I write: ## Building AI systems • Patterns for Building LLM-based Systems: https://t.co/q7ZDvPN7W6 • What We’ve Learned From A Year of Building with LLMs: https://t.co/POueEI2Keu • The First Rule of Machine Learning: Start without Machine Learning: https://t.co/moE92m3eMO • Design Patterns in Machine Learning Code and Systems: https://t.co/FNyhUaBUvp • More Design Patterns For Machine Learning Systems: https://t.co/KFjTIX85qa • System Design for Recommendations and Search: https://t.co/OEbm0tB34i • How to Write Design Docs for Machine Learning Systems: https://t.co/IPfOfFg8fr ## Evals and testing • Task-Specific LLM Evals that Do & Don't Work: https://t.co/Rv3okJ28AL • Evaluating the Effectiveness of LLM-Evaluators: https://t.co/kTKo5T2iag • Out-of-Domain Finetuning to Bootstrap Hallucination Detection: https://t.co/jFCqUMOBqw • How to Test Machine Learning Code and Systems: https://t.co/HCc5DweOQz • Writing Robust Tests for Data & Machine Learning Pipelines: https://t.co/7VHRCtWVRw • Don't Mock Machine Learning Models In Unit Tests: https://t.co/DtnNtMAVQP ## Synthetic data, prompting, attention, etc • How to Generate and Use Synthetic Data for Finetuning: https://t.co/NbSawVfVSg • How to Write Data Labeling/Annotation Guidelines: https://t.co/X5J4VWBZkb • Bootstrapping Labels via ___ Supervision & Human-In-The-Loop: https://t.co/uozLWg5ttK • Prompting Fundamentals and How to Apply them Effectively: https://t.co/tIFaESf8se • Some Intuition on Attention and the Transformer: https://t.co/iqyd6q9SDu https://eugeneyan.com/writing/llm-patterns/ https://applied-llms.org https://eugeneyan.com/writing/first-rule-of-ml/ https://eugeneyan.com/writing/design-patterns/ https://eugeneyan.com/writing/more-patterns/ https://eugeneyan.com/writing/system-design-for-discovery/ https://eugeneyan.com/writing/ml-design-docs/ https://eugeneyan.com/writing/evals/ https://eugeneyan.com/writing/llm-evaluators/ https://eugeneyan.com/writing/finetuning/ https://eugeneyan.com/writing/testing-ml/ https://eugeneyan.com/writing/testing-pipelines/ https://eugeneyan.com/writing/unit-testing-ml/ https://eugeneyan.com/writing/synthetic/ https://eugeneyan.com/writing/labeling-guidelines/ https://eugeneyan.com/writing/bootstrapping-data-labels/ https://eugeneyan.com/writing/prompting/ https://eugeneyan.com/writing/attention/

Good match?

8 yr old video from Andrew Ng about error analysis which applies equally well (if not better) to debugging AI products Ng: "I usually do this in a spreadsheet, but using an ordinary text file would be ok" 70% of evals is looking and counting https://t.co/tS7mYl7Wms https://www.youtube.com/watch?v=JoAxZsdw_3w

Good match?

Overview of the series 1. We’ve been measuring wrong. @beirmug showed that traditional IR metrics optimize for finding the #1 result. RAG needs different goals: coverage (getting all the facts), diversity (corroborating facts), and relevance. Models that ace BEIR benchmarks often fail at real RAG tasks. 2. Retrieval can reason. @orionweller 's models understand instructions like “find documents about data privacy using metaphors.” His Rank1 system generates explicit reasoning traces about relevance. These models find documents that traditional systems never surface. 3. Single vectors lose information. @antoine_chaffin demonstrated how late-interaction models like ColBERT preserve token-level information. No more forcing everything into one conflicted representation. Result: 150M parameter models outperforming 7B parameter alternatives on reasoning tasks. 4. One map isn’t enough. @BEBischof and @loldedxd showed why we need multiple representations. Their art search demo finds the same painting through literal descriptions, poetic interpretations, or similar images—each using different indices. Stop searching for the perfect embedding. Build specialized representations and route intelligently. Annotated slides can be found here: https://t.co/r8DXpCCzS9 35% off our AI Evals course (its our last live cohort): https://t.co/S9ctbeyvCJ https://hamel.dev/notes/llm/rag/not_dead.html http://bit.ly/evals-ai

Good match?

The experiments conducted in this post illustrate how early we are as an industry on eval tooling. Some takeaways and related thoughts: 1. Naively applying automation (which many current frameworks do) is likely to fail. 2. It's easy to get fooled that automation (esp overzealous automation) is giving you valuable insights. Stay skeptical at all times! 3. We have to design eval workflows so human-in-the-loop accelerates effort while helping you externalize what "good looks like" 4. Qualitative analysis hasn't sufficiently made its way into eval tooling as much as it should. There are opportunities to design better automation here. (QA is super underrated for evals btw)

Good match?

We created flashcards for students in our Evals course, but are giving them away for free! First up, Error Analysis - the most important part of evals. More info in the reply

Good match?

fwiw, perhaps I have too many evals: no numbers but basically green good red bad, and we iterate out of reds into more green. and when existing evals saturate (cols 1 - 4), we look at traces and add more evals (col 6 & 7) 🤓 we can think of evals as just the scientific method: observe outputs, annotate them, hypothesize why good or bad, design and run experiments, measure outcomes, analyze errors, repeat. it's possible to do this without evaluation sets or automated evaluators, but it could be hard to scale; also, those eval sets and evaluators are by no means a substitute for trying the product yourself and looking at complaints / defects. anecdotes > metrics. I try to explain evals to engineers by analogizing it to test-driven development (TDD), where we write tests before implementing software that passes those tests. eval-driven development (EDD) has the same philosophy: before developing a feature, we define success criteria (evals) and start measuring against it from day one. machine learning teams have done this for decades really, though we call it validation and test splits. same same but different. i'm probably paranoid and over-obsessive for tracking this on a spreadsheet that goes from column A -> Z, but I don't know of a better way for a team to collaborate across time and space, with everyone running their own experiments and merging changes, and identifying the most promising directions to pursue or existing gaps to close. if you do know, please let me know! 🙏

Good match?

Over the last 2 yrs, I've helped 35+ companies improve their AI products. I distilled my approach into this guide, which covers error analysis, synthetic data, eval-human alignment, involving domain experts, optimizing # of experiments & more (1/5) https://t.co/Kn3hyq5zf2 https://hamel.dev/blog/posts/field-guide/

AlexReibman

@AlexReibman

Good match?

Evals are a scam. And we're being gaslit into believing they aren't. New post just dropped (🧵).

Good match?

The most common mistakes I see teams make with LLM judges: • Too many metrics • Complex scoring systems • Ignoring domain experts • Unvalidated measurements That's why I wrote this guide, w/ detailed examples to help people avoid these issues (1/4) https://t.co/qWoti5QDvO https://hamel.dev/blog/posts/llm-judge/

Good match?

Does anyone know how circleback is so good? Like the AI meeting notes are just uncanny. Nothing else I’ve tried comes remotely close Everyone I know wants to know how they do it https://t.co/AjZXKCYBUR https://circleback.ai/?via=hamel

Good match?

"We can write the best content in the world, but we don't have any distribution." A founder told me this last week. After years of indie consulting and growing my own audience, I wrote about what works and what most people are doing wrong. https://t.co/PfdPi1Mddu 1/5 https://hamel.dev/blog/posts/audience/

Good match?

The bugs I encounter most with LLMs in production are related to data drift. This is acute for LLMs b/c of all the moving parts: prompts, RAG, functions, etc. There is a classic ML technique that works for detecting drift. I explain it in this post: https://t.co/slyxqPD1EW https://hamel.dev/blog/posts/drift

Good match?

(Reposting) New blog post 📰: Understanding what LLM frameworks are doing is difficult (digging through code, docs, etc). I show how intercept API calls with to quickly understand how popular frameworks work and find interesting results. https://t.co/p2KaerHjyo https://hamel.dev/blog/posts/prompt/

Good match?

I wrote some notes about a very **uncool topic** - looking at and curating your LLM data TLDR; I highly recommend building your own data curation/annotation tools if you are working with LLMs. Made some specific tool recommendations here: https://t.co/rXgSRdHPl8 https://hamel.dev/notes/llm/04_data_cleaning.html

Good match?

Here's an engaging intro to evals by @sridatta and @iamwil. They've clearly put a lot of care and effort into it, where the content is well organized with plenty of illustrations throughout. Across 60 pages, they explain model vs. system evals, vibe checks and property-based tests, designing eval criteria, aligning llm-evaluators, how to measure alignment via various metrics, how to analyze evals to improve our system, and more. Now I can just direct folks to https://t.co/ye6tUo8OIz instead of having to write it myself haha https://forestfriends.tech

Good match?

New blog post: Through my LLM consulting, I've had to deploy many demo applications and private websites. I've found one magical tool: @dokku 💫 It's like Heroku, but you own it, and it's been absolutely game-changing. https://t.co/j9Lv5ldrsv https://hamel.dev/blog/posts/dokku/

Good match?

Honestly surprised we still have to repeat this message. h/t @FanaHOVA Repeat after me: I will build evals for my tasks. I will build evals for my tasks. I will build evals for me tasks.

simonw

@simonw

Good match?

This is excellent, really great to see more writing on the topic of evals, especially valuable when it discusses building them for a specific domain My brief notes here: https://t.co/CicemEryu6 https://simonwillison.net/2025/Feb/12/building-a-snap-llm/

zeeg

@zeeg

Good match?

need this stuff to go into codecov+junit specs

lennysan

@lennysan

Good match?

"Evals are emerging as the real moat for Al startups." — @garrytan (YC CEO) "Writing evals is going to become a core skill for product managers." — @kevinweil (OpenAI CPO) "If there is one thing we can teach people, it's that writing evals is probably the most important thing." — @mikeyk (Anthropic CPO) "Evals are surprisingly often all you need." — @gdb (OpenAI President) Writing evals is quickly becoming a core skill for anyone building AI products (which will soon be everyone). Yet there’s very little specific advice on how to get good at this. In today's post, @amankhan tells you everything you need to understand to master this skill, including: 1. WTF are evals 2. Why are they so important 3. The three different eval approaches 4. The four distinct parts of a great eval 5. A workflow for writing effective eval + more Don't miss this one → https://t.co/coexS7y2He https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-complete