EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics

EduFair-Bench is a benchmark for auditing the pedagogical fairness of large language model (LLM) tutors. It tests whether the quality of tutoring varies systematically with students’ demographic backgrounds. It was developed at the Swiss Federal Institute of Technology in Lausanne (EPFL). It addresses a gap in existing tutoring benchmarks, which measure teaching quality but not whether that quality is distributed equitably across student groups.

The benchmark draws on a filtered question bank of mathematics questions from SocraTeach and physics and chemistry questions from SciQ. Each tutor model conducts Socratic, multi-turn dialogues with a fixed simulated student across nine demographic levels in four dimensions: gender, immigration background, first language and socioeconomic status. Because the question and the student stay the same within each comparison, differences can be attributed to how the tutor responds to demographic signals. Two ablations help separate tutor-driven bias from effects caused by the simulated student. One conveys demographics only through a student’s name. The other gives the tutor and the student conflicting demographic information.

Each tutor turn is scored on five pedagogical metrics: mistake recognition, answer leakage, step scaffolding, actionability and corrective tone. Each dialogue is also scored on four conversation-level metrics. Scoring uses an LLM judge validated against human annotators. Bias is measured using paired Wilcoxon signed-rank tests with bootstrapped effect-size confidence intervals. Applied to five tutors, the benchmark found that capability and fairness were largely unrelated. It also found that pedagogy-focused training shifted bias rather than removing it, and that language and immigration cues produced the largest gaps. Data, code, prompts and rubrics are publicly released.

Auto-discovered on 2026-10-14 by OECD Catalogue Automation