AI short-drama studio

Animation: a trace of one three-shot run through the platform. A writers' room plans the story, a grey-box set is built, keyframes are painted, then each shot is scripted and rendered in turn while a continuity memory passes a brief from each shot to the next script. A GPU row shows the language model, the image model and the video model taking turns on one 32 GB card. Finally the film is cut and finished.

PauseDrag to scrublanguage modelimage modelvideo modelCPU only

The problem

Video models generate a few seconds at a time, but a drama needs dozens of shots, and every shot is a separate generation. Faces drift, the room changes at every cut and props jump between hands. On the first test film the store looked different in every shot.

There was a second constraint: everything had to run on one workstation, where the language models, an image model and a video model all need the same 32 GB GPU. I built a platform that plans a drama the way a film crew does, keeps people, sets and props consistent from cut to cut, and schedules every model on that one card. It runs entirely locally, with cloud models as an option rather than a requirement.

How it works

1. 1Plan

   • LLM writers’ room: story, blocking, actors’ notes
   • A sceptical viewer marks fake beats

2. 2Build the set

   • 3D grey-box set from the plan
   • Checked for collisions and blocked doors

3. 3Paint keyframes

   • Assets made in dependency order
   • First frames painted over the grey box

4. 4Write each shot

   • Local LLM script writer
   • A memory brief for this shot only

5. 5Render and cut

   • Video with sound, one shot at a time
   • Timeline, finishing and export

After each take, its last second is read back into the continuity memory, which briefs the next shot.

Engineering the platform

Everything runs on one workstation: an RTX 5090 with 32 GB of video memory and 94 GB of RAM. Most of the engineering went into making the models share it, and into making every run fast, recoverable and measurable.

• Several models, one GPU, taking turns. The video model keeps about 50 GB of weights in pinned RAM and uses up to 30 GB of video memory; the local writer needs about 26 GB. A lease-and-yield protocol hands the card over: ComfyUI unloads, the language-model server measures the real free memory with nvidia-smi, sizes its context to fit, checks that nothing spilled into system RAM and cleans up orphaned servers. Rewriting a script after a render went from 185 s to 40 s.
• A silent slowdown, found and fixed. A 1080p take was running at about 800 s per step instead of 90, because Windows was quietly spilling video memory into system RAM. Launching ComfyUI with a memory headroom brought it back to 89 s per step, and a token budget now downsizes over-long takes and restores them at export.
• Faster renders, measured. An 8-step distilled LoRA, rewritten for a pruned base model, rendered 2.1× faster with sharper detail. SageAttention halved a whole job (24:46 to 12:29), and a CUDA 13 build of PyTorch made each step 2.5× faster on the RTX 5090. The pruned model gives identical frames with 13 GB less video memory.
• A job engine that survives restarts. The Workbench runs one GPU job at a time from a queue that recovers after a crash. Each story is a JSON document with atomic writes, locks while the assistant edits it, snapshots and undo. Every step fingerprints its inputs, so an edit marks exactly the downstream steps as stale and lists them under “Next steps”.
• Local or cloud, per step. Planning, the set, scripts, review, keyframes and images can each run on a local or a cloud model, and a failed cloud step falls back to the local one. Cloud steps receive only visual descriptions, never the story.
• Defaults chosen by experiment. A same-seed A/B harness compares settings against decision rules fixed in advance, with identity, motion, sharpness and audio-clipping metrics and side-by-side reports. That is how a cache accelerator was rejected: it was faster, but it cooled the lighting and made skin look plastic.

The Workbench

Key decisions

• A 3D grey-box set keeps the room the same. Keyframes drawn from text alone couldn’t produce true reverse angles; they just copied the main view. Now the planner lays each location out as a metric 3D set (areas, doors, fixtures, actor marks and a camera per shot), code checks it for collisions, blocked doors and cameras inside walls, and a NumPy rasteriser renders the views that keyframes are painted over. That gives true reverse angles, 5 of 6 keyframes matching the set, and a smaller loudness jump at cuts (3.6 to 2.0 dB).
• Pin only the first frame. Starting each shot from a painted keyframe raised the main-angle layout score from 2.0 to 5.0 (judged by a vision model on a 1–5 scale). Pinning the last frame as well improved the metrics but made the scene morph in the middle of the shot, so only the first frame is pinned, and a probe now checks inside shots as well as at the cuts.
• A continuity memory. A typed ledger records every person, prop and fixture at the start and end of each shot (position, facing, what each hand holds, who knows what), with each value tagged as seen, scripted or inferred. Each new shot gets a brief retrieved for it alone, plus clips of earlier footage that show the fixtures it needs.
• A writers’ room, not a one-shot planner. Separate LLM roles research, write, block, critique and revise the story. In blind pairwise comparisons it beat the one-pass planner in 6 of 8 cases.
• Making a small local LLM reliable by splitting the task. The local writer mixed up who does what to whom. A separate step now fixes each beat as a finished sentence before the script is written, and a validator checks every beat: correct beats went from 60 of 68 to 68 of 68. A custom quote-tolerant grammar stopped dialogue lines being cut off.

Results so far

Frames from the fully local run

Where it started

The first version, ITV (June to July 2026), was a FastAPI platform with its own node canvas, a “film crew” of LLM roles, BM25 retrieval over the story bible and earlier episodes, text-to-speech with lip-sync, and an anime pipeline. Identity drift between shots was its biggest problem, and the lessons from it shaped the platform above.

All people shown are AI-generated. Frames come from shots rendered locally with the open-weight MiniMax H3 model.