Lipsynced music video

The brief

A client wanted a music video for their song: a photorealistic AI singer performing the whole track, in a sensual, deliberately provocative style. Hosted video services refuse that kind of content or tone it down until it is no longer what was asked for. So the project had to run on open models on my own hardware, where no third party’s content policy decides what the video may show.

What you see here

The client’s video was delivered and approved. It is private, and I am not sharing it. The video on this page is a pilot I made with the same pipeline to demonstrate it.

How it was made

I give the art direction: the look, the rooms, the set-ups, the mood of each part of the song, and what I like or don’t like in the results. Everything else is automated. An AI agent runs the whole pipeline: it turns my directions into prompts and pictures, renders and checks every shot, redoes the failures, assembles the edit and finishes the video. It all runs locally in ComfyUI on one RTX 3090 Ti, on a custom node package and director tool I built for it.

1. Song and lyrics. The agent splits the song into a full mix and a vocals-only track and times every sung word against the vocals. The video model obeys this timing over the audio, so it has to be right: automatic alignment alone left 27% of the singing untimed; after the agent’s correction step it was 4%.

2. Reference pictures. I describe the performer, the outfits and the rooms; the agent generates the pictures with Qwen-Image 2.1:

• a front portrait of her
• each outfit on a figure without a head
• each room from the exact angle and lens of the shot that uses it

It models each room as a simple 3D plan and places every shot’s camera in it, so the background always has the right perspective.

3. Shots. From my direction for each part of the song, the agent writes every shot’s prompt (camera first, then her movement) and picks its reference pictures. MiniMax H3 generates the video together with its own soundtrack. Her lips follow that soundtrack, not the song, so the pipeline keeps them on the words three ways:

• a timed singing plan in the prompt
• the vocals muted wherever nothing is sung
• the real vocals locked into the model’s soundtrack

With these, phrases land within 10 ms instead of up to half a second off.

4. Drafts and selection. The agent drafts every shot at low resolution and reviews the results. Contact sheets and automatic checks flag a second face, a face that’s too small, or bars in the frame, and a vision model gives a second opinion. Failed shots are re-rendered; after two failures the agent rewrites the prompt rather than only changing the seed. I make the final call on which takes stay. For 112 cuts in the final edit the agent rendered 614 shots and 1,237 takes.

5. Editing. Every shot sits on the same 24 fps grid of song frames, so cuts between shots are frame-exact and the lips stay in sync on both sides. The agent assembles the edit, and I adjust it on a studio page that works with layers, like a video editor.

6. Finishing. The agent finishes only the takes the edit uses. Each one is scaled up to full HD from the draft’s saved state, so the approved motion and mouth stay the same; the real vocals stay locked and the colour is matched back to the draft. It runs unattended: about 3.5 minutes per cut, roughly 4.5 hours for the whole video. The soundtrack is the original song.