Varest logo

Varest: Just Respect the Semantics

Image-Token Compression via Pre-LLM Semantic Variable-Resolution Tokenization and In-LLM Relevance Pruning

Tianhao Fu1,2,†,* Austin Wang3,† Charles Chen1,4,† Roby Aldave-Garza5 Victor Xie3 Xiyue Jiao1 Yucheng Chen1 Martin Tin1

† Equal contribution · * Project lead

1 University of Toronto2 Vector Institute3 McGill University4 University Health Network5 University of Waterloo

Paper arXiv Code
Video overview From object boundaries and text localization to variable-resolution tokens and learned relevance.

TL;DR Our method preserves nearly lossless performance at less than 25% of the tokens and >85% retention at only 5.56%, by maximizing the semantic fungibility.

Overview

An image is not a uniform grid of information. A coherent object can be summarized; a word or an object boundary needs finer detail. Varest uses this principle of semantic fungibility to decide where to merge, then learns which visual tokens the language model still needs.

Method

Varest combines semantic merging before the LLM with relevance pruning inside it. Early decoder layers see the whole scene; later layers work with a smaller set of visual tokens.

Open PDF
The Varest pipeline Spatial coverage is preserved through semantic merging, before learned [REL] registers select tokens at the decoder midpoint.
01 / Localize

Find meaningful boundaries

Lightweight object and text localizers identify semantic transitions and text regions, guiding where detail should be preserved.

02 / Merge

Use resolution where it counts

Merge compatible regions into variable-resolution tokens. Preserve text at native resolution when the budget allows, while covering the full image.

03 / Prune

Let the model read first

Four learned [REL] registers score visual tokens after early decoder processing, keeping a smaller subset for the remaining layers.

Explore the full architecture
Open PDF
Training across token budgets allows one checkpoint per backbone. Optional adaptive halting removes the remaining image tokens when a prediction-stability and confidence criterion is met.

Visualizations

Explore how semantic structure, text, and scene context are represented as the token budget changes. Select a view below; use “Open PDF” to inspect a figure at full size.

Open PDF
Same scene, different granularity. Object regions become finer as the budget increases. Text tokens, highlighted in orange, remain identifiable across budgets. Examples use LLaVA-1.5-7B.
Open PDF
Keep the text and its context. This example contrasts Varest’s semantic regions with pruning and merging baselines on LLaVA-1.5-7B.
Open PDF
Relevance after reading. Nearly the same tokens are retained for two unrelated questions. The example suggests that early layers can absorb information before the corresponding image tokens are removed.
Inside the tokenizer: object boundaries and text
Open PDF
Object localization. Boundary strength guides region merging.
Open PDF
Text localization. Textness guides the allocation of fine detail.

Results

5.56%

of the layer-averaged image tokens

86.10–92.33%

of full-token performance retained

3 backbones · 13 benchmarks

highest mean retention against 15 baselines at every evaluated budget

Across three LLaVA backbones and 13 benchmarks, Varest achieves the highest mean performance retention among 15 baselines at every evaluated budget. At 5.56% of the layer-averaged image tokens, it retains 86.10–92.33% of full-token performance.

Open PDF
Performance vs. token budget Retention is normalized to each uncompressed backbone; 100% means matching full-token performance.
Open PDF
Performance across visual tasks, backbones, and token budgets.

Performance at a glance

Mean performance retention ↑
Varest performance retention at four layer-averaged image-token ratios.
Backbone5.56% tokens11.11% tokens22.22% tokens38.89% tokens
LLaVA-1.5-7B92.33%97.08%100.37%101.89%
LLaVA-1.5-13B91.31%95.82%100.03%102.37%
LLaVA-NeXT-8B86.10%91.85%96.04%98.18%

Source: Table 1 of the paper. Mean retention averages 15 normalized metrics across 13 benchmarks; POPE contributes three metrics. Adaptive halting is disabled. For LLaVA-NeXT, ratios are applied to each image’s native sequence length.

More performance at half the token budget. On all three backbones, Varest at 11.11% tokens outperforms every evaluated baseline at 22.22%. On LLaVA-1.5-7B, that is 97.08% retention versus the strongest baseline’s 94.50%.

Measured computation and optional adaptive halting
Open PDF
Each point averages performance and computation independently over four budgets. Optional adaptive halting reduces FLOPs by a further 3.0–5.1%, with retention losses of 0.30–0.48 percentage points. These are computation savings, not measured latency improvements.

Discussion

Semantics before selection

Varest preserves full spatial coverage during merging so early decoder layers can integrate scene context. Its learned selection does not require decoder attention weights and remains compatible with FlashAttention.

General visual question answering is relatively robust to compression. Document reading, chart understanding, and referring-expression grounding benefit more from larger budgets.

Scope and open questions

The current evaluation covers three LLaVA backbones. Extending to other model families, including Qwen-VL and InternVL, remains future work.

Lower FLOPs do not automatically imply lower latency; efficient implementation is still needed. Applying semantic fungibility to other domains also requires domain-specific validation.

Where Varest fits among compression methods
Open PDF
Varest combines explicit semantic guidance before the LLM with learned relevance selection inside it.

BibTeX

@misc{fu2026varest,
  title={Just Respect the Semantics: Image-Token Compression via
         Pre-LLM Semantic Variable-Resolution Tokenization and
         In-LLM Relevance Pruning},
  author={Tianhao Fu and Austin Wang and Charles Chen and
          Roby Aldave-Garza and Victor Xie and Xiyue Jiao and
          Yucheng Chen and Martin Tin},
  year={2026},
  eprint={2610.18690},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.18690}
}