CITATION.cff

---
abstract: "This work investigates descriptive captions as an additional source of supervision
  for biological multimodal foundation models. Images and captions can be viewed
  as complementary samples from the latent morphospace of a species, each capturing 
  certain biological traits. Incorporating captions during training encourages
  alignment with this shared latent structure, emphasizing potentially diagnostic
  characters while suppressing spurious correlations. The main challenge, however,
  lies in obtaining faithful, instance-specific captions at scale. This requirement
  has limited the utilization of natural language supervision in organismal biology 
  compared with many other scientific domains. We complement this gap by
  generating synthetic captions with multimodal large language models (MLLMs),
  guided by Wikipedia-derived visual information and taxon-tailored format examples. 
  These domain-specific contexts help reduce hallucination and yield accurate,
  instance-based descriptive captions. Using these captions, we train BioCAP (i.e.,
  BioCLIP with Captions), a biological foundation model that captures rich semantics 
  and achieves strong performance in species classification and text-image
  retrieval. These results demonstrate the value of descriptive captions beyond labels
  in bridging biological images with multimodal foundation models."
authors:
  - family-names: Zhang
    given-names: Ziheng
  - family-names: Ma
    given-names: Xinyue
  - family-names: Campolongo
    given-names: "Elizabeth G."
  - family-names: Thompson
    given-names: "Matthew J."
  - family-names: Zhang
    given-names: Net
  - family-names: Gu
    given-names: Jianyang
cff-version: 1.2.0
date-released: "2025-10-24"
identifiers:
  - doi: "10.5281/zenodo.17437591"
  - description: "The GitHub release URL of tag v1.0.0."
    type: url
    value: "https://github.com/Imageomics/biocap/releases/tag/v1.0.0"
  - description: "The GitHub URL of the commit tagged with v1.0.0."
    type: url
    value: "https://github.com/Imageomics/biocap/tree/32ddb13fc60a80273ae3cf1ddce72113f0cd0f14"
keywords:
  - clip
  - biology
  - CV
  - imageomics
  - animals
  - plants
  - fungi
  - species
  - images
  - taxonomy
  - "rare species"
  - "endangered species"
  - "evolutionary biology"
  - multimodal
  - "knowledge-guided"
  - "synthetic captions"
  - "caption-generation"
license: MIT
message: "If you use this software, please cite both the article and the software itself."
repository-code: "https://github.com/Imageomics/biocap"
title: "BioCAP"
version: 1.0.0
type: software
references:
  - authors:
      - family-names: Gu
        given-names: Jianyang
      - family-names: Stevens
        given-names: Samuel
      - family-names: Campolongo
        given-names: "Elizabeth G."
      - family-names: Thompson
        given-names: "Matthew J."
      - family-names: Zhang
        given-names: Net
      - family-names: Wu
        given-names: Jiaman
      - family-names: Mai
        given-names: Zheda
    cff-version: 1.2.0
    date-released: "2025-09-03"
    doi: "10.5281/zenodo.15644363"
    license: MIT
    repository-code: "https://github.com/Imageomics/bioclip-2"
    title: "BioCLIP 2"
    version: 1.0.1
    type: software
  - authors:
      - family-names: Stevens
        given-names: Samuel
      - family-names: Wu
        given-names: Jiaman
      - family-names: Thompson
        given-names: "Matthew J."
      - family-names: Campolongo
        given-names: "Elizabeth G."
      - family-names: Song
        given-names: "Chan Hee"
      - family-names: Carlyn
        given-names: "David Edward"
    date-released: "2024-09-19"
    doi: "10.5281/zenodo.10895870"
    license: MIT
    repository-code: "https://github.com/Imageomics/bioclip"
    title: BioCLIP
    version: 1.0.2
    type: software
  - authors:
      - family-names: Stevens
        given-names: Samuel
      - family-names: Wu
        given-names: Jiaman
      - family-names: Thompson
        given-names: "Matthew J."
      - family-names: Campolongo
        given-names: "Elizabeth G."
      - family-names: Song
        given-names: "Chan Hee"
      - family-names: Carlyn
        given-names: "David Edward"
      - family-names: Dong
        given-names: Li
      - family-names: Dahdul
        given-names: "Wasila M"
      - family-names: Stewart
        given-names: Charles
      - family-names: "Berger-Wolf"
        given-names: Tanya
      - family-names: Chao
        given-names: "Wei-Lun"
      - family-names: Su
        given-names: Yu
    title: "BioCLIP: A Vision Foundation Model for the Tree of Life"
    year: 2024
    pages: "19412-19424"
    collection-title: "Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"
    type: coference-paper
  - authors:
      - family-names: Ilharco
        given-names: Gabriel
      - family-names: Wortsman
        given-names: Mitchell
      - family-names: Wightman
        given-names: Ross
      - family-names: Gordon
        given-names: Cade
      - family-names: Carlini
        given-names: Nicholas
      - family-names: Taori
        given-names: Rohan
      - family-names: Dave
        given-names: Achal
      - family-names: Shankar
        given-names: Vaishaal
      - family-names: Namkoong
        given-names: Hongseok
      - family-names: Miller
        given-names: John
      - family-names: Hajishirzi
        given-names: Hannaneh
      - family-names: Farhadi
        given-names: Ali
      - family-names: Schmidt
        given-names: Ludwig
    title: OpenCLIP
    version: v0.1
    type: software
    doi: "10.5281/zenodo.5143773"
    date-released: "2021-07-28"