Add new footage

• Plan the shots first: who speaks, what they say, and how each shot leads to the next:
  • a continuous move where the camera would keep rolling (first/last-frame: first frame = the previous shot’s last frame, last frame = the next shot’s first frame; the frame export workflow saves both, seamless-frame-export.json);
  • a motivated cut between speakers;
  • a hard cut only for a jump in time or place.
• The room: a still of the location from the episode, with every person removed, as the base for every shot. In our test a room made from a description didn’t match the episode’s set.
• Speaking shots: the start-image workflow, two takes per line; keep the one whose speech fits. Each clip makes its own voice, so keep a speaker’s lines in as few shots as you can.

Late-1990s live-action television science-fiction scene with film grain, in {THE ROOM, AS IN THE PICTURE}. Close-up of {CHARACTER}, {LIGHTING ON THE FACE}. It (S1), in a {VOICE: soft, calm, even}, says: <d>[English]{THE LINE}</d> Its mouth moves with the words. Static camera. Sound: {THE ROOM'S HUM} under the voice, no music.

Late-1990s live-action television science-fiction scene with film grain, in {THE ROOM}. {WHAT THE PEOPLE DO DURING THE MOVE}. The camera pushes slowly and smoothly in, past {FOREGROUND}, until {CHARACTER} fills the frame in a close-up, about to speak. One continuous camera move, no cut. Sound: {THE ROOM'S HUM}, murmured voices without words that fade to quiet, no music.

• Join: in an editor with separate sound tracks (Shotcut, DaVinci Resolve):
  • trim each shot to its line;
  • drop the doubled frame at a continuous move;
  • L- and J-cuts at speaker changes;
  • a 0.15 s fade-out on each outgoing clip;
  • a muffled room-tone bed at about −12 dB under the dialogue.
• Every piece must be the same size, and each piece’s sound exactly as long as its picture. The join adds sounds end to end, so 24 ms extra per shot put a later scene’s sound 72 ms late in our test.

With ffmpeg instead of an editor

This joins two shots with each clip’s sound at its own picture time (so voices stay on the lips), a 0.15 s fade-out on the outgoing clip, and a low-passed room-tone bed. It was tested with ffmpeg 6.1 and 8.1. For more shots, add each one the way shot 2 is added, with these rules (tested on a 5-shot scene):

• Chained move (its first frame repeats the shot before): trim=start_frame=1 on the picture and atrim=start=0.041708 on the sound. At a cut: no trim on either.
• adelay = the shot’s first frame in the joined scene × 2002 (samples per frame at 48 kHz and 23.976 fps): 248248 for a shot starting at frame 124.
• A fade-out on every clip that has another after it: afade=t=out:st=…:d=0.15, with st = that clip’s own length in seconds (after any trim) minus 0.15.
• The bed: aloop=loop=-1 in place of loop=2 (which covers only about 18 seconds); atrim=0:$D still cuts it to length.
• D = the joined scene’s frames × 1001/24000, and amix=inputs= the number of shots + 1.

FPS=24000/1001
L1=248248          # shot 1's length in samples at 48 kHz: 124 frames × 1001/24000 s × 48000
D=10.301958        # output length: (124 + 123) frames × 1001/24000
ffmpeg -y -i shot1.mp4 -i shot2.mp4 -filter_complex "\
[0:v]setpts=N/($FPS)/TB[v0];\
[1:v]trim=start_frame=1,setpts=N/($FPS)/TB[v1];\
[v0][v1]concat=n=2:v=1:a=0,format=yuv420p[v];\
[0:a]aresample=48000,asplit[a0r][bsrc];\
[a0r]afade=t=out:st=5.02:d=0.15[a0];\
[1:a]aresample=48000,atrim=start=0.041708,asetpts=PTS-STARTPTS,adelay=${L1}S:all=1[a1];\
[bsrc]atrim=1.0:4.0,asetpts=PTS-STARTPTS,lowpass=f=900,asplit[f][r0];[r0]areverse[r];\
[f][r]concat=n=2:v=0:a=1,aloop=loop=2:size=288000,atrim=0:$D,volume=-12dB[bed];\
[a0][a1][bed]amix=inputs=3:normalize=0:duration=longest,apad,atrim=0:$D[a]" \
 -map "[v]" -map "[a]" -r $FPS -c:v libx264 -crf 18 -g 24 -keyint_min 24 -sc_threshold 0 \
 -c:a aac -b:a 192k -movflags +faststart scene.mp4

The bed is cut from shot 1 at 1.0–4.0 s: pick a stretch with no speech, and change atrim=1.0:4.0 to it. The bed is at −12 dB, under dialogue. For a scene with no speech, use volume=-3dB.

• The new footage starts on the episode’s frame and leaves it (the transition is at its end): anchor the start only, with the start-image workflow h3-i2v-first-frame.16gb.json (the episode’s last frame as 200 LoadImage). In our test the camera drifted from the last frame out into the night jungle.
• It comes from somewhere and lands on the episode’s frame (the transition is at its start): anchor the end only, with the last-frame-only workflow. Load the first frame of the shot it leads into in Last frame (the first frame of the shot this transition leads into). In our test a descent through the canopy landed exactly on the next shot’s first frame, in place of a fade.

  Download the last-frame-only workflow (16:9) 4:3 version h3-fl2va-last-frame-only.16gb.json

• Both ends are transitions (a cutaway that fades in and out): neither end is fixed. Use the pictures workflow: the 2.3 workflow without a start frame, with the same picture inputs. In our test it made a mood-matched cutaway of a lantern.

  Download the pictures workflow (16:9) 4:3 version h3-ref2v-pictures.16gb.json

• In the prompt, say how the shot enters or leaves (a descent, a drift, a move away) and how the camera moves (below).

• When the first and last frames don’t show every person and place the scene needs, give the model pictures cut from the episode (the set, the people in it), plus a picture of anyone new, not descriptions.

Download the pictures + start frame workflow (16:9) 4:3 version h3-ref2v-pictures-start-frame.16gb.json

• In our test a new character stepped into the real dinner, matching his picture, and the diners were kept and reacted.
• Pictures of a place must show no people: crop them out, or make an empty still of the set. A still with someone up and moving brings their action into the new shot: in one test the model copied it. With people-free crops of the same place, the people already in the scene stayed still and only the new character moved through.
• In the prompt, say what the people in the scene are doing, and what must not happen: “the diners stay seated and talk quietly; nobody gets up; nobody enters”.
• If the new character must be clearly seen, say where they walk and that they stay fully in frame. In one test the new figure crossed just inside the top edge, half out of frame.
• In those tests the joins were seamless: the start, chained and landing frames were all within about 2–3/255 of their targets, and a chained continuation (2.1) cut back into the original cleanly.