Enhancing Robustness of CLIP to Common Corruptions through Bimodal Test-Time Adaptation

Uloženo v:

Podrobná bibliografie
Vydáno v:	arXiv.org (Dec 3, 2024), p. n/a
Hlavní autor:	Maharana, Sarthak Kumar
Další autoři:	Zhang, Baoming, Karlinsky, Leonid, Feris, Rogerio, Guo, Yunhui
Vydáno:	Cornell University Library, arXiv.org
Témata:	Adaptation Feature extraction Testing time Image enhancement Zero-shot learning Corruption Robustness
On-line přístup:	Citation/Abstract Full text outside of ProQuest
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Popis
Abstrakt:	Although open-vocabulary classification models like Contrastive Language Image Pretraining (CLIP) have demonstrated strong zero-shot learning capabilities, their robustness to common image corruptions remains poorly understood. Through extensive experiments, we show that zero-shot CLIP lacks robustness to common image corruptions at increasing severity levels during test-time, necessitating the adaptation of CLIP to unlabeled corrupted images using test-time adaptation (TTA). However, we found that existing TTA methods have severe limitations in adapting CLIP due to their unimodal nature. To address these limitations, we propose \framework, a bimodal TTA method specially designed to improve CLIP's robustness to common image corruptions. The key insight of our approach is not only to adapt the visual encoders for better image feature extraction but also to strengthen the alignment between image and text features by promoting a stronger association between the image class prototype, computed using pseudo-labels, and the corresponding text feature. We evaluate our approach on benchmark image corruption datasets and achieve state-of-the-art results in TTA for CLIP, specifically for domains involving image corruption. Particularly, with a ViT-B/16 vision backbone, we obtain mean accuracy improvements of 9.7%, 5.94%, and 5.12% for CIFAR-10C, CIFAR-100C, and ImageNet-C, respectively.
ISSN:	2331-8422
Zdroj:	Engineering Database