What is multimodal AI?

Multimodal AI is AI that can work with more than one type of content, such as text, images, audio and video. A multimodal model can read a scanned invoice, describe a photo, transcribe a call or answer a question about a chart in the same conversation. Most leading AI models now accept images and documents as well as text, and many can also produce images or speech.

Why it matters for a business

Business information rarely arrives as clean text. Support tickets carry screenshots, invoices exist as scans, meetings are part speech and part shared screen. Multimodal models take these in directly instead of through a separate conversion step that loses what the layout was carrying. For most businesses, reading documents and images is the useful part, and it is more dependable than generating them. The limits of text models carry over: a misread number comes back as a plausible number in the right place, not as an error.

Example

Let’s say a few hundred supplier invoices sit in a folder as scans. A multimodal model returns the supplier, date and total for each. The ones printed on a regular template come back clean. One photographed page with a handwritten note in the margin comes back with a total that looks ordinary and is wrong.

Go deeper

Multimodal AI: image, audio, and video

Related terms

• Computer vision
• Generative AI
• Large language model (LLM)
• Token

All terms, A to Z