Hallucinations in large language models refer to the generation of content that is not factual, consistent, or grounded in provided context or world knowledge. This phenomenon is often categorized into two types: context hallucinations, where outputs conflict with source material, and extrinsic hallucinations, where outputs are fabricated without basis in pre-training data. The focus here is on extrinsic hallucinations, which require models to be factually accurate and to acknowledge uncertainty when knowledge is lacking.

Causes of Hallucinations

Pre-training Data Issues

Pre-training data, typically sourced from the public internet, may contain outdated, incomplete, or incorrect information. Models learn by maximizing log-likelihood, potentially memorizing inaccuracies, leading to errors.

Fine-tuning with New Knowledge

Fine-tuning models with new data via supervised fine-tuning or reinforcement learning from human feedback can introduce risks. Research indicates that learning from unknown samples is slower and may increase hallucination tendencies compared to known samples. Experiments show that models perform best when fine-tuned primarily on known data with minimal unknown examples.

Detection of Hallucinations

Retrieval-Augmented Evaluation

Benchmarks like FactualityPrompt use Wikipedia as a knowledge base to evaluate hallucinations through metrics such as hallucinated named entity error and entailment ratio. Methods like FActScore decompose long-form outputs into atomic facts for verification against sources, while SAFE employs LLM agents to query search engines iteratively for fact-checking. Frameworks like FacTool standardize factuality detection across tasks including QA and code generation.

Sampling-Based Detection

SelfCheckGPT performs consistency checks by comparing multiple samples from a black-box LLM without external knowledge, using metrics like BERTScore or NLI to identify factual errors.

Calibration for Unknown Knowledge

Benchmarks like TruthfulQA and SelfAware assess model honesty when facing unanswerable questions. Calibration techniques measure output uncertainty, with larger models often better calibrated. Methods like indirect queries test for hallucinated references by asking for auxiliary details, improving detection over direct queries.

Mitigation Strategies

RAG and Attribution

Retrieval-augmented generation provides external context to ground outputs. Approaches like RARR retrospectively edit outputs for attribution using retrieved evidence, balancing attribution and preservation rates. FAVA fine-tunes editors to correct hallucinations based on synthetic error-injected data. Self-RAG integrates retrieval and self-critique tokens during generation to enhance factuality.

Chain of Actions

Chain-of-Verification designs a process where models plan and execute verification steps independently, reducing hallucinations by decomposing verification into short questions. RECITE uses recitation as an intermediate step, leveraging transformer memory to retrieve information before generating answers.

Sampling Methods

Factuality-aware sampling adjusts probability dynamically during token generation to reduce randomness in later parts of sentences, improving factuality. Inference-time intervention identifies attention heads correlated with truthfulness and shifts activations to favor factual outputs.

Fine-tuning for Factuality

Techniques include topic prefix training to enhance fact awareness and sentence completion loss focusing on later tokens. FLAME incorporates factuality into alignment training via supervised fine-tuning and RLHF, using metrics like FActScore as rewards. Factuality tuning applies DPO on samples scored for truthfulness using reference-based or confidence-based methods.

Fine-tuning for Attribution

WebGPT and GopherCite combine web search with fine-tuning to answer questions with verified quotes, using human feedback for reinforcement learning. Models learn to cite sources and reject answers when uncertain, improving factual accuracy.

Evaluation Benchmarks

Key datasets include TruthfulQA for honesty assessment, FactualityPrompt for fact-based evaluation, SelfAware for distinguishing answerable questions, LongFact for long-form factuality, HaDes for binary hallucination classification, FEVER for claim verification, and FAVABench for fine-grained error analysis.