Cross-Modal Consistency Scoring
Description
Cross-modal consistency scoring checks whether content in one modality matches content in another, such as an image and its caption, a generated image and the prompt that produced it, or an audio clip and its description. It uses a contrastively trained model, one that has learned from large collections of paired examples to turn each item into an embedding (a numerical vector describing its content) and to place matching pairs close together: CLIP, trained on image-caption pairs, for images and text, or CLAP, its audio-and-text counterpart, for audio and text. The score for a pair is the cosine similarity of its two embeddings; in the standard image form, CLIPScore, negative similarities are set to zero and the result is rescaled, and no human-written reference is needed. A practitioner scores every pair in an evaluation set or in production, sets a threshold against a sample of human judgements, and then compares average scores between systems or sends pairs below the threshold for review. The score shows whether two items are about the same things, not whether every detail of a description is correct.
Example Use Cases
Reliability
A public library deploying an automatic alt-text generator for its digitised photograph collection scores every generated description against its image and routes those below a threshold, set on a sample rated by staff, to a cataloguer, so that descriptions of the wrong subject are caught before visitors using screen readers meet them; names, dates and places in the descriptions are checked separately.
A design-software company deciding whether to ship a new version of its text-to-image model generates images for the same few thousand varied prompts with fixed random seeds, compares the average image-prompt scores of the old and new versions, and checks the comparison against human ratings on a subset before release.
A team assembling an image-caption training set from web data drops pairs whose image-text similarity falls below a threshold, as the builders of LAION-5B did, and audits a sample of the discarded pairs by subject and demographic group before accepting the filter, because such filters remove some groups' images at higher rates.
A broadcaster's archive team generating text descriptions for an untagged sound-effects library scores each description against its clip with an audio-text model, uses the scores only to order clips for human listening checks rather than to accept descriptions automatically, and records the agreement between scores and listeners' judgements.
Limitations
- Contrastively trained image-text models largely ignore word order, relations between objects and which attribute belongs to which object, so a description that names the right things in the wrong relationship can score almost as well as a correct one. The score checks that the right content is mentioned, not that the description of how it fits together is right.
- It cannot check what is not visible or audible. Names, places, events and other context in a news photograph or an archive recording are outside what the encoder can match, and the original authors found the image score weaker than reference-based metrics on news captions, so it should not be the only check where such facts matter.
- Scores inherit the encoder's training data, which is mostly web alt-text, and its biases. An audit of CLIP-based filtering found that data associated with LGBTQ+ people, older women and younger men was excluded at higher rates, so a single threshold can be stricter for some subjects than others and needs checking by group.
- Scores have no absolute meaning: they depend on the encoder checkpoint and on the tool's scaling (the original CLIPScore multiplies the cosine similarity by 2.5, torchmetrics by 100), so values are not comparable across checkpoints or tools. The default CLIP text encoder also reads at most 77 tokens, so longer descriptions are judged on their opening only. Thresholds must be set and reported for the specific model and settings used.
- The audio form is weaker still: CLAP-based scores have been found to correlate only weakly with human ratings of how well audio matches its text, so for audio the score is a coarse screen to prioritise human checks rather than evidence of consistency on its own.
Resources
Research Papers
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans.This is in contrast to the reference-free manner in which humans assess caption quality.In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M im-age+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references.Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE.Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities.Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation.Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.