Multimodal model
multimodal · vision-language model · VLM
กล่าวโดยย่อ
A multimodal model can process more than one type of input in the same request, most commonly text together with images, and increasingly audio and video. Non-text inputs are converted into the same internal representation as tokens, which is why a single image can consume a substantial share of the context window.
A multimodal model handles inputs of different types in one request — asking about a chart, transcribing and answering from audio, or reading a screenshot of an error message.
Mechanically, non-text input is passed through an encoder that produces vectors in the same space the language model consumes. From the model's perspective there is one sequence; the image simply arrives as a block of positions within it.
That has a direct practical consequence: images consume context budget. A high-resolution screenshot can cost as much of the context window as several pages of text, and the same amount in billing. Downscaling before sending is usually free accuracy-neutral savings for tasks that do not depend on fine detail.
Capability is uneven in ways worth testing rather than assuming. Reading text in an image, describing a scene, and interpreting a diagram are different skills with different reliability. Dense tables, handwriting, and charts where the answer depends on precise value comparison remain the weak spots — the model may produce a confident number that is simply misread.
คำถามที่พบบ่อย
- How much of the context window does an image use?
- It depends on resolution and the provider's encoding scheme, but a full-page screenshot commonly costs on the order of a thousand tokens or more. Check the provider's formula before sending images in bulk.
- Can multimodal models generate images as well as read them?
- Reading and generating are separate capabilities. Many models accept images as input while producing only text; image generation is often a distinct model, sometimes invoked as a tool.