SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Peking University, UC Berkeley, Panasonic Holdings Corporation

SparseVLM+: Visual Token Sparsification with Improved Text-Visual Attention Pattern
Peking University, Tsinghua University

🔥 [2025/12/11] We released new version SparseVLM+! Bring a Stronger Performance with Improved Text-Visual Attention Pattern!

🔥 [2025/06/04] The sparsification code for VideoLLaVA is now open source! Please check the video branch.

🔥 [2025/05/01] Our SparseVLM is accepted by ICML 2025!

🔥 [2025/03/06] We released SparseVLM v1.5! Higher Accuracy, Flexible Pruning Manner, and Compatibility with FlashAttention 2!

🔥 [2024/10/15] We released SparseVLM and its Project Page! The Code is now open-source! Please check the v1.5 branch for the latest version.

• News
• Contents
• Overview
• Preparation
• Usage
• SparseVLM+
• Citation
• Acknowledgment

In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, existing methods extract more compact image representations by modifying the image encoder or projector. While some recent works further sparsify vision tokens during the decoding, they still ignore the guidance from the language tokens, which contradicts the multimodality paradigm. We argue that visual tokens should be sparsified adaptively based on the question prompt, as the model might focus on different parts (e.g., foreground or background) when dealing with various questions, as shown in Figure below. Unlike previous methods with text-agnostic visual sparsification (c) e.g., recent FastV, our SparseVLM (b) is guided by question prompts to select relevant visual patches.

1. Clone this repository and navigate to SparseVLMs folder

2. Install necessary package

3. Download Multimodal Benchmark

Please follow the detailed instruction in LLaVA-Evaluation.

Specifically, setting RETAIN_TOKN in the environment variables indicates the number of tokens to be retained after the SparseVLM algorithm. It supports four numbers of tokens, including 192, 128, 96, and 64. If a specific number of tokens is required, please make modifications in ./llava/model/language_model/score.py

1. Example for evaluating MME results (retain 192 tokens):

2. Example for evaluating TextVQA results (retain 128 tokens):

3. Example for evaluating ScienceQA results (retain 96 tokens):

4. Example for evaluating MMBench results (default 64 tokens):

You can boost the performance of SparseVLM by enabling the V2.0 mode, which can be seamlessly enabled via an environment variable without modifying the code.

1. Example for evaluating MME results (retain 192 tokens):

2. Example for evaluating TextVQA results (retain 128 tokens):

3. Example for evaluating MMBench results (retain 96 tokens):

4. Example for evaluating GQA results (retain 64 tokens):

This project is released under the Apache 2.0 license.

If you use SparseVLM in your research, please cite our work by using the following BibTeX entry:

We extend our gratitude to the open-source efforts of TCFormer, LLaVA, MiniGemini and VideoLLaVA.