FairStress is a 300‑scenario benchmark spanning 29 domains, hosted on Hugging Face, designed to measure how LLMs respond to demographic changes and adversarial prefixes. It integrates directly into Olmo‑eval, so you can run it on any open‑weight model or GPT‑style API with a single command. The dataset includes explicit, implicit, and no‑identity versions of each scenario to test subtle bias.
If you’re evaluating model fairness beyond surface decisions, FairStress gives a more nuanced view than standard parity checks. It does not cover equity‑based fairness or long‑term impact analysis, and its adversarial prefixes are manually curated, so you’ll need to supplement it for deeper coverage. It’s best suited for research labs or teams with evaluation pipelines, not for quick production testing.
Leave a Reply