HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding
Chenxin Tao*, Shiqian Su*, Xizhou Zhu*, Chenyu Zhang, Zhe Chen, Jiawen Liu, Wenhai Wang, Lewei Lu, Gao Huang, Yu Qiao, Jifeng Dai†
*Equal Contribution †Corresponding Author
Published in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025
HoVLE is a high-performance monolithic Vision-Language Model that avoids modality-specific encoders. Unlike existing monolithic VLMs that degrade language capabilities when tuning LLMs for vision, HoVLE introduces a holistic embedding module that converts both visual and textual inputs into a shared space, allowing LLMs to process images the same way as text. The model is trained through a multi-stage strategy: first distilling visual features from a pre-trained vision encoder and text embeddings from the LLM with unpaired data, then performing next-token prediction on multi-modal data, and finally instruction tuning. HoVLE achieves performance comparable to leading compositional models on various benchmarks while substantially outperforming previous monolithic models.