Sound for Film, TV & Games: What You Hear Was Built

Watch a person walk across a wet street in a film and shut their eyes. You will hear rain, but not the rain the camera crew stood in: that was a recording of a different night, laid under the scene. You will hear footsteps, which were not recorded on the street at all; someone watched the picture in a studio weeks later and walked on a patch of gravel in time with the actor. You will hear a distant engine that belongs to no particular road. The voice is probably the only thing that came from the day of filming, and it may have been re-spoken in a booth.

None of this is cheating. It is the craft. A camera shows a rectangle of the world; sound is what makes the rectangle feel like the world, and it can only do that if every layer is chosen, placed and balanced. Then a different question arrives. A game cannot fix a mix in advance, because the player decides when the fight starts. So game audio becomes a small machine that rebuilds the soundtrack as events happen.

This piece is for both readers of the Sound Engineering major: the engineer who will mix for picture, and anyone curious why a film is hard to hear at home. It follows Mixing & Signal Processing (36/63). Everything here is built from the tools in it, aimed at a moving picture. Audio is synthesised in your browser: no recordings are used. Every interactive also shows its numbers, so it works with the sound off.

Dialogue, music, effects: a soundtrack is layers

A finished film mix is built from three families, called stems. Dialogue is every spoken word. Music is the score and any songs. Effects is everything else, and it is the biggest family by far: ambiences (the room tone of a kitchen, the wind of a field), hard effects (a door, a gun, a car-by), and Foley, the human-made sounds of footsteps, cloth and handled objects. The three are kept as separate stems on purpose. When a film is sold abroad, the dialogue stem is replaced with a dubbed one and the music and effects stems, often called the M&E, stay untouched. That is one reason sound is never "one recording".

A sequence of a few seconds can contain dozens of layers. A single punch might be a slap of wet cloth, a low thump from a drum, a short bright snap, and a bit of a bone-crack effect from a library. Sound designers speak of building a sound the way a painter builds a colour: no single layer is the punch, the sum is. Making up a sound that has never been heard, such as a creature or a spaceship, is the same craft with synthesis added (Synthesis & Sound Design, 36/68).

The hardest part of layering is not adding; it is deciding what the viewer can afford to hear. Speech lives mostly between about 300 Hz and 4 kHz, and so do rain, traffic and most of a string section. When they pile up, they mask the words. Mixers fight this by turning down whatever competes with a line while it is spoken, a move called ducking. Try it below.

With all six layers on and no ducking, the voice sits only about 3 dB above the rain, traffic and music added together. Turn ducking on and the gap becomes about 9 dB, with nothing removed from the scene. The viewer still feels the rain; they just understand the line. That is the mixer's job in one move.

Locked to the frame: sync, Foley and ADR

Film runs at a fixed number of pictures a second, and sound has to land on the right one. At 24 frames per second one frame lasts 41.7 ms; at 25 it is 40 ms; at 30 it is 33.3 ms. The eye and ear forgive less than you would expect, and unevenly: sound that arrives after the picture, as it does in nature, is tolerated far longer than sound that arrives before it. We spend our lives hearing thunder after lightning and a shout after the wave of the arm. We never hear a sound before its cause.

Two European and international documents turn this into numbers. The ITU's recommendation BT.1359-1 reported that trained viewers in controlled tests began to notice an error when the audio led the picture by more than about 45 ms, or lagged it by more than about 125 ms. The EBU's recommendation R37 asks that the whole chain, camera to viewer, keep the audio between 40 ms early and 60 ms late of the picture, with each single stage within +5 and −15 ms. Positive means early. The odd shape of the window is itself the lesson.

Because sound is slow, distance alone can break sync. A microphone far from the action records the sound late, which is why the boom is kept close, and why a stadium crowd watching a screen 34 m away hears the screen's audio a tenth of a second behind a perfect feed. Drag the sliders.

Sync is why sound is mostly built afterwards. Production sound, recorded on set, gives the dialogue and a reference of the room. Cameras and dialogue are then cut by the picture editor, and the dialogue editor smooths the cuts with room tone so the background does not jump at every edit. Where a line is unusable (a plane overhead, a noisy wind machine), the actor re-speaks it in a booth while watching their own face, a process called automated dialogue replacement, ADR or "looping". And everything the set microphones missed gets made by a Foley artist, named for Jack Foley (died 1967), who worked at Universal in the transition from silent to sound films, performing footsteps, cloth and handled props in time with the picture. A modern Foley stage has floors of wood, gravel, tile and sand, and a pit of shoes. Foley is the reason a film's footsteps are cleaner and more rhythmic than any real street.

Loudness: the number that decides how you hear it at home

You have turned a film up for a whisper and down for a gunshot. This is not your television; it is a delivery problem. A cinema is a calibrated room: film mixers work at a fixed monitor level so that what they hear is what the cinema plays. At home, the same mix is replayed in kitchens and on laptops, and the mix often covers too wide a range for the room.

Broadcasters had a related problem: advertisements that were far louder than the programmes around them. The cure was to measure loudness the way listeners experience it, over the whole programme, in LUFS (also written LKFS, the same measurement under another name), the unit from Mastering & Loudness (36/64). The European recommendation, EBU R 128 (first issued in 2010, revised several times and most recently in November 2023), asks for an average programme loudness of −23 LUFS, with a tolerance of ±0.5 LU and a maximum true peak of −1 dBTP. The American practice, ATSC A/85, recommends −24 LKFS, within ±2 dB where no metadata is used, measured on the dialogue as the anchor. In the United States the CALM Act made the commercial-loudness rules binding, with FCC rules in force from 13 December 2012. Streaming services publish their own delivery targets; a typical drama specification sits in the −27 to −24 LUFS range.

The measurement has two gates. Anything below −70 LUFS is treated as silence. Then the meter drops everything more than 10 LU below the running average, so very quiet passages do not count. That makes the loud scenes pull the number up, and a loud film is turned down as a whole, leaving its dialogue quieter. Explore it with three scenes.

Default numbers: the dialogue is at −24, the quiet scenes at −40 and the action at −15. The quiet scenes fall below the relative gate and vanish; the rest averages to −20.2 LUFS. To hit −23 LUFS the whole programme must come down by 2.8 dB, so the dialogue ends at −26.8. Under the American rule the dialogue itself is measured and already sits on −24, so nothing moves and the action stays 9 dB above it. Neither rule is wrong; they answer different questions, and both leave the film's own loudness range, the gap between whisper and bang, in the artist's hands. The EBU does not set a maximum loudness range for programmes, and it points out that producers may narrow it for a given playback setting.

Placing a sound in a room: channels, downmix and objects

A cinema soundtrack has speakers all round you. The long-standing home and cinema layout is 5.1: left, centre and right behind the screen, a left and right surround at the sides and back, and a low-frequency effects channel (the ".1"). The ITU's standard (BS.775) puts left and right 30° either side of centre, and the surrounds at 100° to 120°. Dolby Atmos (first used in a cinema for Brave in June 2012) added the idea of objects: a sound is stored with a position, not with a speaker, and the cinema's renderer works out which of its speakers to use. A cinema Atmos track can carry up to 128 tracks, 118 of them objects and the rest a 9.1-channel bed.

The simplest way to place a sound between two speakers is to share it. A sound exactly between them gets 0.707 of its level from each, which is −3 dB, so that the total power stays the same as if it sat on one speaker. That is the constant-power rule, and it is what the demo below does between each neighbouring pair on the ring.

Most people hear the film in two channels, so every mix also has to survive a downmix. The standard stereo downmix adds the centre and the surround to the front pair, each at 0.707 (−3 dB). Watch what the stereo downmix does to a sound as you move it round the room.

At 0° the stereo downmix gives 0.707 in each ear: the centre dialogue lands in the middle, 3 dB down. At 180° the sound shares the two rear speakers at 0.707 each and then loses 3 dB again: 0.5 in each channel, −6 dB. A sound that was loud behind you is quiet in the downmix, which is why careful mixers listen to the stereo fold-down and keep dialogue in the centre channel, where it survives anything.

Games: music that waits for the bar line

A film plays once, in order. A game never does. The player may stand still for ten minutes or start a fight in three seconds, and a score recorded as one track will sit wrong either way. Game composers instead write music as parts that can be combined and switched. Two techniques dominate. Vertical layering stacks parts that can be faded in and out over a loop: a calm pad, then a bass, then drums as danger rises. Horizontal re-sequencing jumps between whole sections. Both rely on a state machine: the game sends the music a state ("explore", "combat"), and rules decide when and how the music follows.

The key rule is when. A change on an arbitrary beat sounds like a stumble; the same change on the next bar line sounds planned. The idea goes back to LucasArts, where composers Michael Land and Peter McConnell built iMUSE in the early 1990s, first used in Monkey Island 2: LeChuck's Revenge, so that music followed what was on screen and changed smoothly. Today's middleware, Wwise and FMOD in particular, expose the same idea as states, switches and "transition on next bar". Play with it.

Games: a world that is simulated, not recorded

In a game, sound sources are objects in a scene, and the engine computes what the listener hears. Three effects do most of the work. Distance: a sound loses level as you move away; the standard model is the inverse-distance law, gain = 1 ÷ distance (distances beyond a reference of 1 m), which is −6 dB for every doubling. Direction: your ears are about 17 cm apart, so a sound from the side reaches one ear before the other. The arrival difference, the interaural time difference, is at most about 0.66 ms for an average adult head, and it is the main cue to direction below roughly 1.5 kHz. Above that, the head shadows the far ear, producing a level difference. A head-related transfer function (HRTF) packs both cues and the filtering of the outer ear into a pair of filters, so that headphones can place a sound in front, behind or above. Occlusion: a wall between you and a source muffles it, removing the highs and lowering the level.

Games also fight repetition. A hundred identical footsteps sound like a machine, so a footstep is a container: several recordings, picked at random, each with small random changes of pitch and level. The engine caps how many voices play at once and drops the quietest, because hardware and ears both run out. Try the positional demo, with headphones if you have them.

What a graduate just sees

• A soundtrack is three stems, dialogue, music and effects, and almost every sound in a scene was chosen and added after filming.
• Levels add as energy, so the gap between the voice and everything else is what decides whether words are understood. Ducking buys it cheaply.
• Sound that arrives after the picture is forgiven; sound before it is not. About 40 ms early or 60 ms late is the working edge, and one frame is about 40 ms.
• Sound is slow: 3 ms per metre. Distance alone shifts sync and, in a game, is something to simulate.
• Loudness is measured over a programme with gates, so loud scenes can make the dialogue quieter. Different rules measure different things.
• A speaker-based mix is a rendering of positions; a good mix survives the stereo fold-down, and dialogue lives in the centre.
• Game music is a state machine whose timing rules matter as much as the notes.
• Game sound is a calculation: distance, direction, occlusion, with randomness to hide repetition.

Where it shows up next

• Live Sound Reinforcement (36/66): the same distance, delay and level arithmetic, with the audience as the room.
• Mastering & Loudness (36/64): the LUFS meter used here is the one streaming services use for music.
• Synthesis & Sound Design (36/68): creature voices, engines and interface sounds are synthesised, not recorded.
• Composition, Arranging & Orchestration (35/72): scoring to picture and writing adaptive music are composing under timing constraints.
• In the world: streaming dialogue-clarity settings, hearing-accessible subtitles and audio description, VR, podcasts with sound design, and every cinema trailer.

If you want to go deeper

• 📘 Textbook. Tomlinson Holman, Sound for Film and Television (Focal Press, third edition 2010), the standard course text on production sound, editing, mixing and the technical side of delivery.
• 📘 Textbook. Karen Collins, Game Sound (MIT Press, 2008), the history and theory of game audio; and Andy Farnell, Designing Sound (MIT Press, 2010), which builds effects procedurally.
• 📘 Gentler. Michel Chion, Audio-Vision: Sound on Screen (Columbia University Press, 1994), short and full of ideas about how sound changes what we think we see. For a documentary, Making Waves: The Art of Cinema Sound (2019, directed by Midge Costin).
• 🌐 Free course. Audiokinetic's Wwise-101 certification program: seven lessons and quizzes, free to study; and the EBU R 128 page, which links the free loudness documents.