Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

Guardado en:

Bibliografiske detaljer
Udgivet i:	arXiv.org (Nov 28, 2023), p. n/a
Hovedforfatter:	Leng, Sicong
Andre forfattere:	Zhang, Hang, Chen, Guanzheng, Li, Xin, Lu, Shijian, Miao, Chunyan, Bing, Lidong
Udgivet:	Cornell University Library, arXiv.org
Fag:	Hallucinations Vision Training
Online adgang:	Citation/Abstract Full text outside of ProQuest
Tags:	Tilføj Tag Ingen Tags, Vær først til at tagge denne postø!

Beskrivelse
Resumen:	Large Vision-Language Models (LVLMs) have advanced considerably, intertwining visual recognition and language understanding to generate content that is not only coherent but also contextually attuned. Despite their success, LVLMs still suffer from the issue of object hallucinations, where models generate plausible yet incorrect outputs that include objects that do not exist in the images. To mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential causes of object hallucinations. This adjustment ensures the generated content is closely grounded to visual inputs, resulting in contextually accurate outputs. Our experiments show that VCD, without either additional training or the usage of external tools, significantly mitigates the object hallucination issue across different LVLM families. Beyond mitigating object hallucinations, VCD also excels in general LVLM benchmarks, highlighting its wide-ranging applicability.
ISSN:	2331-8422
Fuente:	Engineering Database