Cross-Modal Drift

Cross-modal drift is the reason you cannot trust a description of an image just because it sounds fluent. It is the difference between a system that merely reads well and one that actually sees. Understanding it gives you the ability to audit any multimodal AI — a captioning system, a visual assistant, an agent that acts on what it 'sees' — and to build systems that verify what they claim. It unlocks further study in hallucination mitigation, multimodal grounding, and evaluation, which you already have notes on: Object Hallucination, Multimodal Grounding Failures, and Content Moderation Guardrails.

Imagine you ask a friend to describe a photo they have not seen. They would produce a description that sounds entirely plausible — 'a sunny park with children playing' — but it is not about the actual photo. Cross-modal drift is exactly that: a model generates output that is consistent with the kind of input it has seen before (the training distribution) but not with the specific input you gave it. It is subtle because nothing about the output looks wrong. The words are correct, the grammar is perfect, the details are ordinary. Only a fact-check against the input reveals the mismatch. In multimodal systems, the input is typically an image plus a text prompt. The model's job is to describe what is in the image. But the model has learned patterns from millions of images and captions, and when the image is ambiguous or the language priors are strong, the model falls back on what it has seen before, not what is in front of it. The output drifts away from the true contents of the image, even though it drifts toward what is statistically likely. That is the drift: the gap between what the model says and what the input actually contains.

Think of a tour guide who has memorised a script about a city but is asked to give a tour of a specific house. The guide knows the city's general layout, the typical architecture, the common features of houses in that area. But the house they are shown has a peculiar feature — say, a bright purple door. The guide, relying on the script, says 'and here you see the standard oak door', because that is what is typical. The description is plausible for the city, but it is wrong for this house. The guide has drifted from the actual house to the general pattern. That is cross-modal drift. The model does not have a script in the same way, but it has learned statistical associations. When the image contradicts the statistical norm, the norm often wins. The analogy works well because it captures the core: fluency without faithfulness. It breaks down because the tour guide could, if asked, look again and correct themselves. A neural network cannot easily 'look again' — it processes the input once, and its output is already determined. It also cannot reason about what it sees; it just generates text that is probable given the input. So the guide analogy understates how persistent the drift is. The model will confidently assert the wrong thing without any awareness of the error.

Cross-modal drift is the production of output that is plausible for the training distribution as a whole but not for the specific input provided, because the model relies on learned language priors rather than the actual non-text evidence.

You have notes on hallucination mitigation, and cross-modal drift is one of its failure modes in multimodal systems. It is closely related to two neighbouring topics you already have notes on: Object Hallucination (the model describes entities not in the image) and Multimodal Grounding Failures (the broader class of outputs making claims the non-text evidence does not support). Cross-modal drift is distinct from object hallucination in that it is not limited to missing objects — it can be about attributes, relationships, spatial layout, or any detail. It is also related to content moderation guardrails, because a guardrail that checks whether the output is grounded in the input is a way to catch drift.