HaRiM+

HaRiM+ is a reference-free evaluation metric that assesses the quality of generated summaries by estimating the hallucination risk within the summarization process. It uses a modified summarization model to measure how closely generated summaries align with the original text, emphasizing the likelihood of hallucinations—statements that aren’t supported by the input. The HaRiM+ model introduces a regularization term into the objective function to capture hallucination risk, which provides insight into the reliability of generated content.

Formula:

In HaRiM+, the quality score of a summary is calculated based on the token likelihood of the summary relative to the original input text. The formula involves a regularization term for hallucination risk:

1. Quality Score (Q):

Quality Score = Sum of Token Log Likelihood - Hallucination Risk Term

2. Hallucination Risk Term (R):

This regularization term captures the likelihood of unsupported content in the summary. It’s computed based on the divergence between the generated tokens and likely tokens derived from the input text, adjusted by an empty-source encoder-decoder.

Higher scores in HaRiM+ indicate better summary quality, as they imply lower hallucination risk and higher alignment with the source content.

Example Usage and Applications:

• Summarization Evaluation: HaRiM+ provides a reference-free method to evaluate summary quality, making it particularly useful for real-world applications where reference summaries may be unavailable.

• Hallucination Detection: By measuring hallucination risk, HaRiM+ helps identify summaries with unsupported or potentially misleading information, ensuring greater factual alignment.

• Model Development: Developers can use HaRiM+ to fine-tune models, minimizing hallucination and enhancing the factual accuracy of generated content.

Impact:

HaRiM+ improves trustworthiness in text summarization by offering a way to gauge factual consistency without needing reference summaries. This metric can be instrumental in applications such as news summarization, content generation for educational materials, and document summarization in legal or medical domains where accuracy is critical.

Son, S., Park, J., Hwang, J., Lee, J., Noh, H., & Lee, Y. (2022). HaRiM+: Evaluating Summary Quality with Hallucination Risk. arXiv preprint arXiv:2211.12118.