Robust Uncertainty Quantification for Factual Generation of Large Language Models

This paper addresses the critical limitation of hallucination in Large Language Models (LLMs) by proposing a novel and robust uncertainty quantification method (RU) for factual generation. It constructs a set of "trap questions" with fake names to evaluate LLM dependability in real-world applications requiring critical thinking. The study highlights that traditional uncertainty quantification methods are deficient when confronted with non-canonical or adversarial questioning strategies, emphasizing the need for more robust approaches to ensure reliability and trustworthiness of AI-generated content.