Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency

Bill Psomas¹†, Dionysis Christopoulos²†, Eirini Baltzi², Ioannis Kakogeorgiou⁶
Tilemachos Aravanis¹, Nikos Komodakis^3,4,5, Konstantinos Karantzalos², Yannis Avrithis, Giorgos Tolias¹

¹Visual Recognition Group, FEE, Czech Technical University in Prague ²National Technical University of Athens ³University of Crete ⁴Archimedes, Athena RC ⁵ACM-FORTH ⁶IIT, NCSR “Demokritos”

Official PyTorch implementation and benchmark results for Efficient Probing.

TL;DR: We introduce efficient probing (EP), a lightweight multi-query cross-attention mechanism that improves accuracy of frozen pretrained encoders while yielding interpretable attention maps.

Overview

As fine-tuning becomes impractical at scale, probing is emerging as the preferred evaluation protocol. However, standard linear probing can understate the capability of models whose pre-training optimizes local representations rather than an explicit global representation. This motivates attentive probing, an alternative that uses attention to selectively aggregate patch-level features. Despite growing adoption, attentive probing is still underexplored: existing approaches are often over-parameterized and computationally inefficient.

In this work, we revisit attentive probing through the lens of the accuracy vs. parameter-efficiency trade-off. We present the first comprehensive study of existing methods, analyzing their design choices and benchmarking their performance. Building on these insights, we propose efficient probing (EP), a lightweight yet effective multi-query cross-attention mechanism that eliminates redundant projections and reduces the number of trainable parameters. Across multiple benchmarks and pre-training paradigms, EP consistently outperforms linear probing and previous attentive probing methods, and remains effective when combined with parameter-efficient fine-tuning. Beyond evaluation, our analysis uncovers emerging properties of EP, including complementary attention maps, which open new directions for leveraging probing beyond protocol design.

Benchmark

Top-1 accuracy of linear probing (LP) vs. efficient probing (EP) on frozen encoders.

This table is meant to grow. If you evaluate a backbone we have not covered, please open a pull request adding a row — see Contributing a row.

Sorted by EP. Ties broken by LP.

#	Family	Method	Arch.	Pre-training	Evaluation	Image size	LP	EP
1	VLM	SigLIP2	SO400M/14	WebLI	IN-1K	224	—	87.68
2	Hybrid	DINOv3	ViT-L/16	LVD-1689M	IN-1K	224	86.6	87.1
3	VLM	SigLIP2	ViT-L/16	WebLI	IN-1K	256	85.2‡	87.0
4	VLM	SigLIP	ViT-L/16	WebLI	IN-1K	256	84.1‡	86.1
5	GEN	AIMv2	ViT-L/14	custom	IN-1K	224	84.8‡	85.9
6	Hybrid	DINOv2	ViT-L/14	LVD-142M	IN-1K	224	85.2	85.6
7	Hybrid	DINOv3	ViT-B/16	LVD-1689M	IN-1K	224	84.0	84.4
8	Hybrid	Franca	ViT-L/14	IN-21k	IN-1K	224	83.8	84.3
9	Hybrid	DINOv2	ViT-B/14	LVD-142M	IN-1K	224	83.2	84.0
10	MIM	CAPI	ViT-L/14	IN-1K	IN-1K	224	81.5	83.6
11	VLM	CLIP	ViT-L/16	WIT	IN-1K	224	82.3	83.4
12	MIM	BEiTv2	ViT-B/16	IN-1K	IN-1K	224	79.0	81.7
13	MIM	MAE	ViT-L/16	IN-1K	IN-1K	224	76.0	79.3
14	Hybrid	iBOT	ViT-B/16	IN-1K	IN-1K	224	78.7	79.2
15	JEA	DINO	ViT-B/16	IN-1K	IN-1K	224	77.3	77.8
16	MIM	MAE	ViT-B/16	IN-1K	IN-1K	224	67.7	75.6
17	JEA	BYOL	RN-50	IN-1K	IN-1K	224	74.3	75.1
18	MIM	SimMIM	ViT-B/16	IN-1K	IN-1K	224	51.5	65.1
19	MIM	MAE	ViT-S/16	IN-1K	IN-1K	224	47.4	64.6
20	GEN	DiT	DiT-XL/2	IN-1K	IN-1K	256	32.7‡	57.0

Grouped by family (same rows, ordered by paradigm)

Family	Method	Arch.	Pre-training	Image size	LP	EP
MIM	CAPI	ViT-L/14	IN-1K	224	81.5	83.6
MIM	BEiTv2	ViT-B/16	IN-1K	224	79.0	81.7
MIM	MAE	ViT-L/16	IN-1K	224	76.0	79.3
MIM	MAE	ViT-B/16	IN-1K	224	67.7	75.6
MIM	SimMIM	ViT-B/16	IN-1K	224	51.5	65.1
MIM	MAE	ViT-S/16	IN-1K	224	47.4	64.6
JEA	DINO	ViT-B/16	IN-1K	224	77.3	77.8
JEA	BYOL	RN-50	IN-1K	224	74.3	75.1
Hybrid	DINOv3	ViT-L/16	LVD-1689M	224	86.6	87.1
Hybrid	DINOv2	ViT-L/14	LVD-142M	224	85.2	85.6
Hybrid	DINOv3	ViT-B/16	LVD-1689M	224	84.0	84.4
Hybrid	Franca	ViT-L/14	IN-21k	224	83.8	84.3
Hybrid	DINOv2	ViT-B/14	LVD-142M	224	83.2	84.0
Hybrid	iBOT	ViT-B/16	IN-1K	224	78.7	79.2
VLM	SigLIP2	SO400M/14	WebLI	224	—	87.68
VLM	SigLIP2	ViT-L/16	WebLI	256	85.2‡	87.0
VLM	SigLIP	ViT-L/16	WebLI	256	84.1‡	86.1
VLM	CLIP	ViT-L/16	WIT	224	82.3	83.4
GEN	AIMv2	ViT-L/14	custom	224	84.8‡	85.9
GEN	DiT	DiT-XL/2	IN-1K	256	32.7‡	57.0

Grouped by backbone scale

Scale	Family	Method	Arch.	Image size	LP	EP
Small	MIM	MAE	ViT-S/16	224	47.4	64.6
Base	Hybrid	DINOv3	ViT-B/16	224	84.0	84.4
Base	Hybrid	DINOv2	ViT-B/14	224	83.2	84.0
Base	MIM	BEiTv2	ViT-B/16	224	79.0	81.7
Base	Hybrid	iBOT	ViT-B/16	224	78.7	79.2
Base	JEA	DINO	ViT-B/16	224	77.3	77.8
Base	MIM	MAE	ViT-B/16	224	67.7	75.6
Base	MIM	SimMIM	ViT-B/16	224	51.5	65.1
Large	VLM	SigLIP2	SO400M/14	224	—	87.68
Large	Hybrid	DINOv3	ViT-L/16	224	86.6	87.1
Large	VLM	SigLIP2	ViT-L/16	256	85.2‡	87.0
Large	VLM	SigLIP	ViT-L/16	256	84.1‡	86.1
Large	GEN	AIMv2	ViT-L/14	224	84.8‡	85.9
Large	Hybrid	DINOv2	ViT-L/14	224	85.2	85.6
Large	Hybrid	Franca	ViT-L/14	224	83.8	84.3
Large	MIM	CAPI	ViT-L/14	224	81.5	83.6
Large	VLM	CLIP	ViT-L/16	224	82.3	83.4
Large	MIM	MAE	ViT-L/16	224	76.0	79.3
Other	JEA	BYOL	RN-50	224	74.3	75.1
Other	GEN	DiT	DiT-XL/2	256	32.7‡	57.0

Paradigms: MIM masked image modelling · JEA joint-embedding architectures · Hybrid MIM + JEA · VLM vision-language models · GEN generative models.

Notes.

• All numbers are top-1 accuracy at the best epoch, not the final one.
• EP is the best result over a sweep of query counts Q (EP_Q in the paper). The best Q is not constant across backbones — it is usually 32, but larger values win for some (e.g. 128 for DiT). Compare rows with this in mind.
• For the Hybrid methods, both --cls_features ep (patch tokens) and --cls_features ep_all (patch + [CLS]) were evaluated and the better one is reported, which is ep_all. Other rows use ep.
• Image size is the evaluation resolution. It is not constant — SigLIP, SigLIP2 and DiT run at 256, the rest at 224 — so rows at different resolutions are not perfectly like-for-like.
• Provenance. Rows are full 90-epoch runs unless noted. SigLIP2 SO400M/14 was run with --early_stop: it peaked at epoch 6 and stopped at 29, so the extra epochs would not have helped, but it is not a literal 90-epoch run. Its LP is still being measured.
• LP is the better of the [CLS] token (--cls_features cls) and global average pooling over patch tokens (--cls_features pos). ^‡ marks rows where GAP was used, either because the encoder has no [CLS] token (DiT, AIMv2) or because it already applies an attention pooling of its own (SigLIP, SigLIP2), making its pooled output an unfair stand-in for [CLS].

Contributing a row

1. Run LP and EP on your backbone (see Experiments). Keep the protocol fixed: 90 epochs, LARS, --blr 0.1, effective batch size 4096.

2. LP — report the better of --cls_features cls and --cls_features pos. If the encoder has no usable [CLS], use pos and mark the value with ^‡.

3. EP — sweep --ep_queries (32 is a good starting point; try 8/16/64/128 too) and report the best. Also try --cls_features ep_all alongside ep, and report whichever wins.

4. Report the best-epoch accuracy.

5. Add one line to results.csv and regenerate the tables — never edit the README tables by hand, they are derived:

   python tools/gen_leaderboard.py        # rewrites the README block from results.csv
   python tools/gen_leaderboard.py --check # verifies the README is in sync (used in CI)

   family,method,arch,pretrain,eval,image_size,lp,lp_gap,ep
   Hybrid,MyModel,ViT-L/14,LVD-142M,IN-1K,224,85.0,no,86.2

   lp_gap is yes when LP used global average pooling instead of [CLS]; the script adds the marker for you, sorts by EP, and rebuilds all three views.

6. Open the PR with the winning Q, whether it came from ep or ep_all, and a link to the training log.

Emerging Properties

We jointly visualize the attention maps of EP₈. An emerging property of EP is that its queries specialize in different object regions, yielding complementary and interpretable attention patterns. Queries consistently attend to distinct parts, producing stable semantic correspondences (e.g., tails, beaks, feet) across images and a structured decomposition of visual cues.

Environment

pip install -r requirements.txt

Optional extras, needed only for specific backbones: open_clip_torch (CLIP/SigLIP), diffusers (DiT/SiT), aim (AIMv2).

Important

timm must stay at the pinned 0.9.16. From timm 1.0.x onwards, VisionTransformer passes scale_attn_norm to block_fn, which the custom Block in models_vit.py does not accept, so every models_vit backbone fails at construction. Installing open_clip_torch will silently upgrade timm — reinstall the pin afterwards.

Integration (drop-in EP)

Use Efficient Probing (EP) as a lightweight attentive pooling over patch tokens from a frozen backbone (e.g., ViT). EP learns a small set of queries, attends to tokens with a single key projection, uses identity values (no V/O projections), and averages per-query outputs into one descriptor. It returns both the pooled descriptor and interpretable attention maps.

from poolings.ep import EfficientProbing
# ---- Minimal integration example ----
# In your model.__init__:
   self.ep = EfficientProbing(dim=embed_dim, num_queries=32)  # EP_32

# In your model.forward(...):
#  'tokens' are the outputs of a FROZEN backbone (e.g., ViT):
#  shape (B, 1+N, D) if a [CLS] token exists, else (B, N, D)
#
#  Use only patch tokens (default in our paper/code):
   patch_tokens = tokens[:, 1:, :]          # or 'tokens' if you have no [CLS]
#
#  Optional: include [CLS] among the values by passing all tokens:
#  patch_tokens = tokens                    # uncomment to include [CLS]
#
   pooled = self.ep(patch_tokens)           # pooled: (B, D)
   logits = self.head(pooled)               # your classifier head

Notes

• Freeze the backbone; train only EfficientProbing and your classification head.
• num_queries controls speed/accuracy (e.g., 8, 16, 32). EP averages across queries, so the output stays (B, D).
• Inputs & shapes: tokens are (B, N, D) or (B, 1+N, D) if a [CLS] token exists.
• Default usage: pass patch tokens only (tokens[:, 1:, :] when [CLS] is present).
• To include [CLS] among values, pass all tokens instead.
• Outputs: pooled is (B, D) for your head; optional attn is (B, Q, N) for visualization/analysis.
• Repro tip: set seeds to make the learned query initialization reproducible.

Experiments

Evaluating MAE ViT-B with Efficient Probing on ImageNet-1k:

torchrun --nproc_per_node=4 --nnodes=1 \
    main_linprobe.py --amp bfloat16 --num_workers=12 --dataloader_affinity_hack \
    --epochs=90 --accum_iter=1 --optimizer=lars --batch_size=1024 \
    --model vit_base_patch16  --finetune vit_base_patch16_224.mae \
    --dataset_name imagenet1k --nb_classes 1000 --data_path /path/to/imagenet_pytorch \
    --output_dir ./outputs/linprobe_mae_vitb_ep_imagenet1k \
    --cls_features ep --ep_queries 32

• To perform standard linear probing (LP):

  • Use --cls_features cls to utilize the class token from the pre-trained model.
  • Use --cls_features pos to utilize the patch tokens (via global average pooling).

• --ep_queries sets the number of EP queries (EP_Q in the paper), e.g. 8, 16, 32. Default: 32. The pooled descriptor stays (B, D) regardless, so only the query bank grows.

• To perform full finetuning (FT), use the --finetuning flag.

• Early stopping (optional). --early_stop ends a run once validation accuracy plateaus, rather than always training the full --epochs. Handy for large encoders, where the last tens of epochs often buy very little. Tune with --early_stop_patience (epochs without improvement, default 5), --early_stop_min_delta (accuracy gain that counts as progress, default 0.05), and --early_stop_min_epochs (never stop before this, default 15). Off by default, so the standard protocol is unchanged.

  [!WARNING] Be conservative with these. Under the default cosine schedule, validation accuracy keeps improving until close to epoch 90, so there is no strong plateau to detect. Replaying 14 completed runs, patience=5, min_delta=0.05, min_epochs=15 stopped at a median epoch 49 and cost up to 7.7 points on the worst run, whereas patience=8, min_delta=0.05, min_epochs=30 cost at most 0.27 points but saved only ~4% of the compute. A stopped run is not strictly comparable to a full 90-epoch one.

🎯 More poolings, Please!

• Supported attentive pooling methods (as described in the paper): abmilp, simpool, clip, siglip, aim, ep, cbam, coca, cait, dinovit, jepa, dolg, cae
  • These can be passed via the --cls_features argument.
  • Note: Appending the suffix _all to any pooling type (e.g., ep_all) will include both patch tokens and the class token as input to the selected attentive pooling. By default, only patch tokens are used.

🌐 More datasets, Please!

• Experiment with more datasets in any setup of your choice by adjusting the --dataset_name, --nb_classes, and --data_path arguments accordingly.
  • Supported datasets: ImageNet-1k, Places365, CIFAR-100, StanfordCars, Food101, FGVCAircraft, SUN397, DTD, OxfordIIITPet, CUB200

🛠️ More models, Please!

• Try CAPI and DINOv2 pre-trained models (from PyTorch Hub) by adjusting the --model argument based on their official repositories.

  • The --finetune argument is not needed in this case.

• Try SimMIM, BEiTv2, and iBOT by passing the checkpoint path to the --finetune argument.

  • Pretrained weights are provided via Google Drive.

• Instructions on how to use pre-trained models from OpenCLIP are provided in the following subsection.

Evaluating CLIP ViT-L (pre-trained by openai) with Efficient Probing on ImageNet-1k:

torchrun --nproc_per_node=4 --nnodes=1 \
    main_linprobe.py --amp bfloat16 --num_workers=12 --dataloader_affinity_hack \
    --epochs=90 --accum_iter=1 --optimizer=lars --batch_size=1024 \
    --model ViT-L-14 --openclip_pretrain openai --openclip \
    --dataset_name imagenet1k --nb_classes 1000 --data_path /path/to/imagenet_pytorch \
    --output_dir ./outputs/linprobe_clip_openai_vitl_ep_imagenet1k \
    --cls_features ep --ep_queries 16

• To evaluate alternative pre-trained OpenCLIP models, adjust the --model and --openclip_pretrain arguments accordingly. Available combinations can be found in the official OpenCLIP repository.

  Example alternative:

  --model ViT-L-16-SigLIP-256 --openclip_pretrain webli --openclip

Acknowledgments

This codebase is based on the official MAE, SimMIM and Beyond [cls] implementations.

We thank the authors for open-sourcing them.

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Citation

If you find this repository useful, please consider giving a star 🌟 and citation:

@inproceedings{
psomas2026attention,
title={Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency},
author={Bill Psomas and Dionysis Christopoulos and Eirini Baltzi and Ioannis Kakogeorgiou and Tilemachos Aravanis and Nikos Komodakis and Konstantinos Karantzalos and Yannis Avrithis and Giorgos Tolias},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=PXo0gtT7Al}
}