The July 13, 2025 paper " Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications" introduces a practical **framework for evaluating safety risks** in real-world Large Language Model (LLM) applications, arguing that current methods focusing only on foundation models are inadequate. This framework consists of two main parts: **principles for developing customized safety risk taxonomies** and **practices for evaluating these risks** within the application itself, which often includes components like system prompts and guardrails. It emphasizes the need for organizations to **contextualize general risks** and create taxonomies that are practical and specific to their operational context, as demonstrated by a case study from a government agency. The document then outlines a **safety testing pipeline** that involves curating meaningful and diverse adversarial prompts, running automated black-box tests, and evaluating model responses, particularly focusing on the use of refusals as a measure of safety.

Source:

July 13, 2025

Measuring What Matters: A Framework for Evaluating Safety Risks

in Real-World LLM Applications

https://arxiv.org/pdf/2507.09820