Factual Correctness

Factual Correctness is a metric designed to rigorously evaluate the alignment of a generated response with an authoritative reference, ensuring the factual integrity of AI outputs. It measures factual accuracy, capturing the extent to which the model’s response reflects verifiable claims presented in the reference material. This metric is essential for applications requiring high reliability and accuracy, particularly in domains where misinformation or unsupported statements could have significant consequences. By quantifying factual alignment, Factual Correctness facilitates a detailed examination of the model’s adherence to reference information, supporting improved response quality in AI systems.

To compute Factual Correctness, the process involves:

1. Claim Decomposition: Segmenting both the generated response and the reference text into discrete, assessable claims.

2. Natural Language Inference (NLI): Employing NLI to verify the factual overlap between corresponding claims in the response and reference.

3. Metric Calculation: Using the identified True Positives (TP), False Positives (FP), and False Negatives (FN) to calculate precision, recall, and F1 score.

Precision, recall, and F1 scores provide a comprehensive assessment of factual overlap and are computed as follows:

• Precision: TP / (TP + FP) — Indicates the proportion of claims in the response that are factually supported by the reference.

• Recall: TP / (TP + FN) — Measures the completeness of the factual content relative to the reference.

• F1 Score: 2 × (Precision × Recall) / (Precision + Recall) — Balances precision and recall to reflect the overall factual alignment.

Factual Correctness - Ragas Documentation