Current multimodal information retrieval studies mainly focus on single-image inputs, which limits real-world applications involving multiple images and text-image interleaved content. In this work, we introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences, and the model is required to understand the semantics from the interleaved context for effective retrieval.

We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.

To explore the task, we adapt several off-the-shelf retrievers and build a dense baseline by interleaved multimodal large language model (MLLM). We then propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity, to address the challenge of excessive visual tokens in MLLM-based TIIR models.

Experiments demonstrate that simple adaption of existing models does not consistently yield effective results. Our MME achieves significant improvements over the baseline by substantially fewer visual tokens.

The dataset has been open-sourced on hf-link. Please download it and place it in the data folder.

  CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch --num_processes=8 --mixed_precision=bf16 \
    ./train/train.py \
    --output_dir $output_dir \
    --qrels_path $qrels_path \
    --doc_path $doc_path \
    --query_path $query_path \
    --doc_image_root $doc_image_root \
    --model_name_or_path $model_path \
    --do_train \
    --num_train_epochs $num_train_epochs \
    --per_device_train_batch_size 4 \
    --bf16 \
    --learning_rate $lr \
    --warmup_ratio 0.1 \
    --dataloader_num_workers 4 \
    --logging_steps 50 \
    --do_lora true \
    --gc_query_chunk_size 1 \
    --gc_doc_chunk_size 1 \
    --overwrite_output_dir

• Embedding Generate

  <!-- MME -->
    python ./src/gen_embeddings_ds.py \
      --out_path $out_path \
      --model_name_or_path $base_model_path \
      --checkpoint $checkpoint \
      --image_token_num $image_token_num \
      --qrels_path data/qrels.jsonl \
      --doc_path data/docs.jsonl \
      --query_path data/queries.jsonl \
      --doc_image_root data/doc_images/ \
      --batch_size 8 \
      --maxtoken 4096 \
      --mixed_precision bf16 

  <!-- Two-stream Models -->
    python ./src/gen_embeddings_clip.py \
      --out_path $out_path \
      --model_name_or_path $model_name_or_path \
      --qrels_path data/qrels.jsonl \
      --doc_path data/docs.jsonl \
      --query_path data/queries.jsonl \
      --doc_image_root data/doc_images/ \
      --batch_size 16 \
      --it_type $it_type \
      --image_type $image_type \
      --model $model \
      --mixed_precision fp16 

  <!-- Single-image Multimodal Models -->
    accelerate launch --num_processes 1 ./src/gen_embeddings_multigpu.py \
      --out_path $out_path \
      --model_name_or_path $model_name_or_path \
      --qrels_path data/qrels.jsonl \
      --doc_path data/docs.jsonl \
      --query_path data/queries.jsonl \
      --doc_image_root data/doc_images/ \
      --batch_size 4 \
      --model $model \
      --mixed_precision bf16 

  <!-- Text Models -->
    python ./src/gen_embeddings_text.py \
      --out_path $out_path \
      --model_name_or_path $model_name_or_path \
      --qrels_path data/qrels.jsonl \
      --doc_path data/docs.jsonl \
      --query_path data/queries.jsonl \
      --doc_image_root data/doc_images/ \
      --batch_size 64 \
      --model $model \
      --maxtoken $maxtoken \
      --mixed_precision bf16 

• Retrieval

    python ./src/retrieval.py   \
      --query_embed_path  $out_path/query_embedding_${model}.pkl \
      --doc_embed_path $out_path/doc_embedding_${model}.pkl \
      --out_path $out_path \
      --model $model \
      --dim $dim

• Release code for model training and evaluation.
• Release code for evaluation.
• Release code for data curation.
• Release the wikiHow-TIIR dataset.

Our retrieval corpus is built opon wikiHow-VGSI.