« Little Mean Things | Main | Television (Soul Man) »

June 09, 2006

Single-screen sync|sound

Working with a single screen is essentially like creating an installation, but only having one TV and one set of speakers to work with. Since the viewer can't walk around to control the mix of sounds, or stand in front of one TV to focus on one sound-producing image at a time, I have to find other ways of ensuring she can see each sound.

Part of the challenge for single-screen sync|sound has to do with inherent differences in sound and vision. With music, all the instruments play at the same time, and all the sounds are mixed together. In an installation, this isn't a problem, as the 20 TVs produce a visual equivalent to what is happening with the sound.

However, with a single screen, there is no visual equivalent to instrumentally layered music. You can hear 20 instruments mixed together harmoniously, and you can see them mixed together as 20 TVs in a room, but you can't easily see 20 images at once on a single screen.

One can do half-disolves to mix images together, but with more than two images, it becomes noisy very quickly. The viewer may not be able to distinguish any images at all. Similarly, I can divide the single screen into a grid and shrink the images into each square, but it still doesn't approximate the experience of an installation of multiple TVs, even if you blow it up on a giant screen. It's not immersive. The viewer can't move through a space, he can only look at a single screen.

To get sync|sound to work - with either single or multiple screens - it seems essential that the viewer is able to organize the sounds and images. The viewer needs to be able to link individual sounds with individual images in such a way that it's obvious that a particular image is producing a particular sound, and the sound and image are two parts of a whole, ie. an audio|video sample.

Additionally, a viewer needs to be able to distinguish one audio|video sample from another. In normal music, it may not matter whether a listener can pick out a bass from a guitar or a piano. But with sync|sound, it's essential to the construction of a piece. It needs to be apparent that a music video is composed out of a bunch of disparate samples mixed together.

When all the samples have to share a single screen, each sample ends up fighting to be seen. If we translate a 20 TV installation to a single screen, the viewer is likely to get about 20 times less information. It's much more difficult for the viewer to recognize which image goes with which sound with a single screen. And she no longer has the luxury of seeing a single sample looping, visually isolated from the others.

As a result, single screen sync|sound is much more challenging for both the viewer and the composer. If I'm not careful with the samples I choose, it becomes impossible to get the sync sound effect. The images seem disconnected from the music. The sound and image split into two distinct tracks that don't necessarily even seem related to each other. It looks like music with video wallpaper in the background.

To make things a bit simpler when working with a single screen, I ended up restricting samples to ones where you see an action and hear the resulting sound. This might be a stick striking a drum, a finger plucking a string or pressing a key, someone singing, someone talking, and so on. I would avoid a violin|building sample. I would avoid a piano sample where the image showed the piano player's face. So not only does a sample have to have sync sound, it should also have an action that can be seen to be producing the sound. Without this, it's just too difficult for the viewer to figure out what's going on.

This also explains why so many of the samples are people playing musical instruments.

Although a multi-screen presentation is easier and probably provides the best possible viewing experience, single screen presentation is more democratic. Anyone can view the video, at any time, anywhere. And you can do more with it. You can watch it on your computer, play it on your TV, add it to your music collection, watch it mixed with other music videos, put it on your video iPod - whatever you like. The only thing you can't do is walk through it.

Posted by AK_Alias at June 9, 2006 10:10 PM

Comments

Post a comment




Remember Me?