Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

Salvato in:

Dettagli Bibliografici
Pubblicato in:	arXiv.org (Nov 28, 2023), p. n/a
Autore principale:	Leng, Sicong
Altri autori:	Zhang, Hang, Chen, Guanzheng, Li, Xin, Lu, Shijian, Miao, Chunyan, Bing, Lidong
Pubblicazione:	Cornell University Library, arXiv.org
Soggetti:	Hallucinations Vision Training
Accesso online:	Citation/Abstract Full text outside of ProQuest
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Descrizione
Abstract:	Large Vision-Language Models (LVLMs) have advanced considerably, intertwining visual recognition and language understanding to generate content that is not only coherent but also contextually attuned. Despite their success, LVLMs still suffer from the issue of object hallucinations, where models generate plausible yet incorrect outputs that include objects that do not exist in the images. To mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential causes of object hallucinations. This adjustment ensures the generated content is closely grounded to visual inputs, resulting in contextually accurate outputs. Our experiments show that VCD, without either additional training or the usage of external tools, significantly mitigates the object hallucination issue across different LVLM families. Beyond mitigating object hallucinations, VCD also excels in general LVLM benchmarks, highlighting its wide-ranging applicability.
ISSN:	2331-8422
Fonte:	Engineering Database