Multimodal AI: image, audio, and video

Image generation, video, voice and reading a scanned document are sold as four different technologies. They are one machine. A model predicts the next chunk of whatever it has been given, and only the kind of chunk changes: words, patches of a picture, slices of sound, frames. Knowing which chunk is in play tells you where each one fails.

The same machine underneath

A model is a pattern-matching machine that predicts the next chunk, shaped by what came before and by the prompt.

For text, the chunks are tokens: roughly words, sometimes word-pieces.

For images, the chunks are patches: small compressed stand-ins for regions of the picture. Pixel by pixel would be far too slow.

For audio, the chunks are slices of sound at very short intervals, or compressed stand-ins for them.

For video, the chunks are frames, or pieces within frames stitched across time.

Change the chunk and the same loop produces a different medium. That is what multimodal means when a vendor says it: one model that handles more than one kind of input, and sometimes more than one kind of output.

Making images

A model given a prompt does not draw. It starts with a field of noise, image-shaped and containing nothing, and refines it in passes. Each pass asks what the text loop asks: given the prompt and what is here already, what should the next version look like? Early passes settle the big things, where the mass sits, where the light comes from, roughly what shape an object is. Later passes settle the small ones, the grain of the wood, the highlight on the glass, the letterforms on the label. After 20 or so passes the noise is a photograph of something that has never existed. The model can do that because it was trained on hundreds of millions of captioned images, so it knows what “jar” and “morning light” look like as patterns.

Broad shape first and fine detail last is also why the failure modes are what they are.

It has never seen the specific thing. The model knows jars in general, not one company’s jar, its cap, its label, its red. Showing it photographs of the real product first does work, and it is a small project rather than a prompt.

Nothing carries between two runs. Broad shape is settled fresh each time, so the same product generated twice is two similar products. Tooling has grown up around pinning a product or a face to a reference photograph, which turns consistency into a workflow step with a setup cost.

Words on a label come out right at short lengths. In-image text used to be reliably garbled and is not any more. It was not fixed by teaching the model to spell: it still draws letters as shapes, settled late with the other fine detail. A three-word headline is dependable. A paragraph of ingredients copy, text curving round a lid, or a script the training data barely contained still slips.

Anatomy improved on the same terms. An ordinary pose is fine. An unusual one, a hand doing something specific, a limb foreshortened, still breaks.

So image generation is solid wherever a close-enough answer is fine, and brittle wherever a specific person, product or set of words has to come out exactly. Disclosure is now an obligation rather than a coming one: in the EU, AI-generated images, audio and video have to be disclosed and marked in machine-readable form. Other jurisdictions are moving the same way.

Example

Let’s say a packaged-foods company of about 300 people is putting a new range of cooking sauces into retail, and marketing wants concept visuals for the buyer pitch. Someone types a prompt: a jar of tomato and chilli sauce on a wooden kitchen counter, morning light, space on the right for copy. What comes back shows a buyer what the shelf could look like. It is not the company’s jar, the label reads as a label rather than as the real ingredients panel, and a second run produces a similar jar rather than the same one.

Reading images and documents

Reading is the other half of vision, and the half that pays for itself faster. The model reads an image and produces text about it: same loop, other direction, patches in and tokens out.

That one capability pulls fields out of receipts and invoices, reads charts with checking because small errors on dense data are common, writes image descriptions for accessibility, and spots defects in product photos.

How far it gets depends on the documents. One supplier, printed, same template every month, clean scan: close to solved. Difficulty climbs with layout variety across suppliers, dense tables, especially merged cells and line items running across pages, handwriting in the margins, poor or rotated scans, and scripts thinly represented in training.

The failure shape matters more than any accuracy figure. A model that cannot read a number does not return a blank or an error. It returns a plausible number: right format, right place on the page, wrong. That is the “approximately right” problem with a currency symbol in front of it, and the reason a document workflow needs a check on the totals rather than a check on whether it ran.

For most everyday business work this is the vision capability that carries the load. It is more practical, more reliable and present in more real workflows than image generation, and it gets a fraction of the attention.

Example

Let’s say a few hundred supplier invoices sit in a folder as scans, none of them keyed in. The model reads them and returns supplier, number, date and total for each. The invoices from the regular suppliers, printed on the same template every month, come back clean. One photographed page with a handwritten note in the margin comes back with a total that looks perfectly ordinary and is wrong. That is why someone checks the totals against the purchase orders, rather than checking that the job ran.

Voice and audio

“Voice AI” gets used as if it were one thing. It is at least three.

Speech-to-text. Audio in, text out. Mature: reliable for major languages, decent across most accents, weaker on overlapping speakers and background noise. Meeting notes, call transcripts, dictation. In practice a solved problem.

Text-to-speech. Text in, audio out, now near-indistinguishable from a person. Voice cloning, a synthetic voice built from a few minutes of someone’s speech, is routine and carries real consent and disclosure obligations.

Voice agents. Audio in, audio out, with a model in the middle. A voice agent is an agent with a phone line attached. The classic build stitches three pieces together: speech-to-text, then a language model (an LLM) working out what to do, then text-to-speech. Newer end-to-end voice models do it in one, audio straight in and audio straight out, faster and more natural because the model hears tone rather than a flattened transcript. In the stitched build every piece is swappable, so a better language model is a switch. In the single model it is baked into the weights, and upgrading means retraining the whole thing.

Voice agents work now in bounded flows with a defined job: appointment booking, stock and order questions, password resets, reading back a balance or a delivery window, outbound reminders and surveys. The boundary has moved outward. A working deployment routinely runs several steps in one call, reaching into business systems for the answer (see tools and memory), booking the callback, handing off to a person when it should.

Where they still break:

• Open-ended conversation across many topics.
• Emotional cues, frustration, sarcasm, distress, that a person picks up instantly.
• Several speakers at once, or the phone being handed around.
• Strong accents, noisy environments, fast topic-switching.
• Knowing when to escalate, and doing it gracefully.

The work in a voice deployment is mostly not the voice. Even a narrow one needs a route to a human that works first time, transcripts read through the early weeks, and a decision about what happens when the line mishears a quantity. That is where the project time goes. The line between bounded and open-ended keeps moving outward each release cycle.

Example

Let’s say the support line for that sauce range keeps taking the same question: which sizes are in stock. A voice agent answers the call, looks the answer up in the stock system, reads it back, and passes anything else to a person. The stock question is the easy part. Most of the build is the handover, making sure a caller who asks something different reaches somebody straight away, and deciding what happens when the line mishears a quantity.

Video

Video is the youngest of the four, and two very different things sit under the name.

Video understanding. A model watches a clip and describes, transcribes or summarises it. Working well for short clips, with the usual asterisks on long-form content and dense action.

Video generation. A model produces a new clip from a text prompt, a still image, or a reference clip. It has crossed into usable and quality jumps every few months. A single generated take is short. Length comes from stitching takes together, extending from the last frame of the one before, and stitching is where it drifts: the counter changes shade between shots, the label shifts, the hand holding the jar gains a finger.

Physics is the other limit, and the mechanism behind it makes the rest predictable. The model is producing plausible next frames. Nothing inside it holds the idea that objects continue to exist, so anything whose correctness depends on continuity through time, sauce pouring, cloth, smoke, a crowd, hands doing something precise, is where it comes apart, and the longer the take the worse it gets. Lip-sync splits the same way: speech synchronised to a generated mouth is now a strength of the leading models, while keeping the same person recognisably the same person across several shots is the harder half.

Good for short stylised social clips, internal explainers, background motion, and pre-visualisation before a real shoot. Not ready for long-form photorealism, anything where continuity has to hold across many seconds, or work competing head-on with a real production.

Example

Let’s say the same team wants a short clip for social. A few seconds of the jar on a counter, with light moving across it, works. The shot they asked for, sauce pouring over pasta and held for 10 seconds, comes back with a pour that behaves like nothing that exists, and with the jar changing shade each time one take is stitched to the next.

One model, several kinds of input

Frontier models, the top tier from the big labs, take more than one kind of input at once. Paste an image into a chat and ask about it. Upload a PDF mixing text and charts and the model reads both. Some take audio directly.

That matters wherever information naturally arrives in several forms: a support ticket with screenshots attached, a document mixing prose and diagrams, an invoice that exists only as a scan, a meeting that is part transcript and part shared screen. The whole thing goes in as one prompt, instead of a pipeline converting everything to text first and throwing away what the layout was carrying.

Output is going the same way: a reply with a generated chart inside it, a deck assembled from a prompt, a call that pushes something visual to the screen mid-conversation. Input is further along than output. For anything being decided now, mixed input is dependable and mixed output still wants a look before it reaches a customer.

What carries across all four

The limits from the fundamentals transfer almost line for line.

• The same hallucination problem. A fabricated logo, an invented label, a wrong face, a transcribed word nobody said. Same mechanism, the model producing what fits the pattern, different surface.
• The same context limits. There is finite room for image, audio or video input. Hour-long video, multi-channel audio and very high-resolution images push against ceilings, and the workaround is the one used for long text: split the input into pieces and summarise, with the same trade-offs.
• The same approximately-right problem. A 90%-correct image is great for a mood board and useless for a logo. The acceptable margin depends entirely on the job.
• The same no-business-context problem. The model has not seen the specific product, the specific customers or the brand voice. It is generic until it is given the company’s own data.
• The same training-data dependence. The model is what it ate. Thinly represented cases, a regional accent, a niche industrial process, an unusual product, drop in quality exactly as they do in text.

None of this is a different kind of AI. The concept visual, the invoice, the phone call and the clip are one machine with the chunk swapped, which is why the same questions settle all of them: what is the chunk, what was the model trained on, where does it fail, and is approximately right good enough here. Whatever medium arrives next will answer to those.