HyFL-CLIP is a hyperbolic fine-tuning framework that makes CLIP robust to long, compositional text descriptions (well beyond CLIP's 77-token limit). Standard CLIP relies on absolute positional encoding and short-caption training, so its image–text alignment degrades when long captions are reordered, condensed, or partially dropped. HyFL-CLIP attributes this brittleness to the Euclidean contrastive objective, which enforces a rigid one-to-one match and has no mechanism to represent hierarchical semantic structure.

Instead of matching in Euclidean space, HyFL-CLIP lifts CLIP embeddings into hyperbolic (Lorentz) space, where the exponential volume growth naturally encodes part–whole and summary–detail hierarchies among summarized tokens, short text, long context, and images.

HyFL-CLIP fine-tunes a CLIP/Long-CLIP backbone with three key ingredients, all implemented on the Lorentz model of hyperbolic space (see model/lorentz.py):

• Cross-manifold similarity distillation. Alignment knowledge from the pretrained Euclidean CLIP is distilled into the hyperbolic student, transferring well-calibrated similarities while the geometry is changed.
• Hierarchical semantic modeling. Summarized tokens, short captions, long-form descriptions, and images are embedded in a shared hyperbolic space so that more complete descriptions entail their partial counterparts.
• Einstein midpoint aggregation. Part-level (token/phrase) representations are combined into whole-level representations via the Lorentzian centroid / Einstein midpoint (centroid, centroid_batch in model/lorentz.py), giving a geometry-consistent way to pool hierarchical features.

Core hyperbolic operations used throughout include the exponential/log maps at the origin (exp_map0, log_map0), pairwise Lorentzian distances, and parallel transport.

HyFL-CLIP improves long-text cross-modal retrieval and, notably, is far more stable under textual perturbations (reordering, summarization, partial deletion), reporting up to a 19.5% improvement in long-text cross-modal retrieval under such perturbations. Evaluation spans long-context cross-modal retrieval, retrieval under caption perturbations, intra-modality retrieval, and short-text cross-modal retrieval. The learned representations also integrate with downstream generators such as Stable Diffusion XL.

Please see the paper for the full quantitative tables.

HyFL-CLIP builds on Long-CLIP and a Lorentz-geometry backbone. Prepare a Long-CLIP-compatible environment (PyTorch + torchvision), then activate it. The provided scripts assume a conda environment named hyflclip:

conda activate hyflclip

Training is driven by train/train.py (distributed via torchrun). The convenience launcher train.sh sets the environment and starts a multi-GPU run:

bash train.sh

Edit train.sh to match your machine (it currently sets CUDA_VISIBLE_DEVICES and --nproc_per_node). Key arguments of train/train.py:

Checkpoints are written to output/ckpts/<exp_name>/ep=<N>_hyfl.pt. For training data preparation, follow the Long-CLIP instructions.

Two HyFL-CLIP checkpoints are available:

• HyFL-CLIP_ViTB.pt: HyFL-CLIP with a ViT-B backbone
• HyFL-CLIP_ViTL.pt: HyFL-CLIP with a ViT-L backbone

The checkpoints can be downloaded from the 🤗 Hugging Face model repo using huggingface_hub:

from huggingface_hub import hf_hub_download

vit_b_checkpoint = hf_hub_download(
    repo_id="jeeit17/HyFL-CLIP",
    filename="HyFL-CLIP_ViTB.pt",
)

vit_l_checkpoint = hf_hub_download(
    repo_id="jeeit17/HyFL-CLIP",
    filename="HyFL-CLIP_ViTL.pt",
)

print("ViT-B checkpoint:", vit_b_checkpoint)
print("ViT-L checkpoint:", vit_l_checkpoint)

Cross-modal retrieval evaluation is in eval/retrieval/. Point the script at a trained checkpoint and run:

bash eval.sh

Under the hood this calls:

python eval/retrieval/unified_eval.py \
    --ckpt_path /path/to/output/ckpts/<exp_name>/ep=1_hyfl.pt \
    --datasets urban1k docci dci sharegpt4v

Supported --datasets: urban1k, docci, dci (Long-DCI), sharegpt4v. Update the dataset image_root / json_path fields near the top of eval/retrieval/unified_eval.py to your local dataset locations:

1. Urban1k, COCO, Flickr30k — see Long-CLIP for downloads.
2. Long-DCI — see TULIP.
3. DOCCI — see DOCCI.

This project builds on Long-CLIP and hyperbolic representation-learning code (Lorentz-model operations). We thank the authors of these works.

The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.

If you find HyFL-CLIP useful, please consider citing:

@misc{jang2026hyflclip,
      title={HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding},
      author={Ji Ha Jang and Hayeon Kim and Chulwon Lee and Junghun James Kim and Se Young Chun},
      year={2026},
      eprint={2607.00428},
      archivePrefix={ECCV},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.00428},
}