The evolution of Digital Art - Part 2: VJs

In the late 1990s, when the DJs already were very popular, another subculture
of visual digital art called the VJs (similar to DJ standing for Disc-Jockey, VJ stands
for Video-Jockey or Visual-Jockey) raised.

Whereever there is a visual output medium (e.g. a big screen in a club or a LED wall at an opendoor gig), a VJ - whose job is filling the screen with nice visuals - is not far away. VJing can be done with various tools and equipment. In the simplest case this would just be a video player. In a more interesting setup, various videosources (among them computers producing visual output) are mixed together with videomixers (e.g. the famous Edirol V4.)

With the growing performance of PC Hardware, various VJ softwares have been written, and many of them claim to be all-in-one-solutions to create, playback, mix and even record all kind of visuals - which is according to my own experience nearly never true, or at least not on a satisfying level.

Basically, a typical VJ-software features many parameters to select and switch between:

1. Resources: images, movies, 3d objects/scenes, generative patterns
   


2. Transitions: change from one resource to the other in a more or less spectacular way
   


3. Effects: changing colors/shapes/geometry/3d transformation etc.
   


4. Mixmodes: overlaying different layers, e.g. a movie in the background with a 3d object in the foreground
   

Most VJ softwares have a sophisticated GUI that allows to access many predefined parameters directly, but since most VJ softwares also allow to access the parameters over specified inputs (typically MIDI) many VJs prefer to control the parameters with third-party equipment. Popular and relatively cheap but good third party MIDI devices are e.g. BCF2000 or Faderfox.

Although the combination of VJ Software on a MAC/PC and an external MIDI device can be understood as the VJ-standard and already results in interesting possibilities, such a setup suffers by the following drawbacks (based on my very own VJ experience during the last 6 years):

1. The amount of parameters of the MIDI device are usually quite limited. While good VJ softwares usually feature hundrets of parameters, most mobile MIDI devices usualy only allow to access a limited amount of banks with a maximum of 128 parameters (MIDI channels) simultaneously. Even with enough MIDI channels in several banks, the mapping is a cumbersome process. Dedicated MIDI devices, especially created to fit with the parameters of a corresponding VJ Software, such as e.g. the ReACT VJ Controller from Netzteil (designed to work well together with the VJ Software Resolume) overcome this problem.
   

2. When the MIDI protocol was designed, it was not intended to be used for VJ applications. Therefore, a MIDI channel usually only has a resolution of 7 bits. In order to understand the consequences, let's analyze what happens if we map a MIDI channel to a 'Rotation' parameter of a VJ software (which rotates a 2d image between -180° and +180°). Assume 'Rotation' was intended to allow smooth rotations even in high resolutions (1600x1200 pixels). The necessary precision of such a rotation is a minimum delta of about 0.1°. This means that for a smooth rotation over 360°, at least a 12bit signal would be needed. But MIDI has only 7bits, which means that the smallest possible change of the value of the MIDI channel will result in a rotation-jump of 2.8°! This means that a smooth continous rotation cannot be done slower than 70°/second (or, in other words: about one full rotation every 5 seconds). Any kind of slower rotation will look like a hicking framerate below 25 frames per second (independant of the real FPS). So, MIDI is pretty useless for a 1:1 mapping between MIDI channels and VJ software parameters. How do VJs deal with this? Well, most of them don't bother about it and simply got used to it. Some VJs argue that they act in smoky clubs, when not only a bigscreen but also all kind of lights are flashing, and probably 90% of the halfstoned audience does not remark drawbacks in smooth parameter changes anyways (and their visuals are stored in low resolution anyways)... ...but I also know other VJs who work in other environments (e.g. at exhibition stands of Mercedes) where quality matters!

Nearly every VJ Software claims to have the best performance, most effects supported, easiest to use GUI etc. Obviously, not everybody can be number 1. I don't want to make a ranking list here, but there are some simple things to be said about performance, effects, GUIs, automation etc., and I would like to explain them using VisualJockey (developed in 1999-2003) as a reference.

Performance
Talking about performance unfortunately means talking about bottlenecks. The typical performance bottleneck of any VJ Software consists in calculations that are done with the CPU rather than the GPU, and thus must access system memory (system RAM) rather than memory on the GFX adapter. As soon as there is a memory transfer from GFX memory to system memory and back, this becomes the ultimate bottleneck - the higher the pixelresolution the more obvious the drop of the framerate. The rendering kernel of VisualJockey is a highly optimized system memory engine, and most of its plugin are MMX optimised (64bit, which means that always 2 32bit pixels are processed simultaneously, even on old (and non officially) 64bit systems!) to the max. Before the aera of pixelshader version 2.0, VisualJockey was one of the best performing VJ systems: it could decode up to about 4 layers of Movies (PAL resolution, = 720x576@25fps), and using several tricks, it could add lots of effects and also mix all layers together (all with about 30FPS, on a PIV 3GHz). It does (except for some additional plugin packages) not use the GPU at all, which means that it just runs fine with cheap old GFX adapters( without any 3d chips needed at all). What really matters on such a system is a fast CPU and a big and fast L2 cache - this commonly applies to all system-memory based VJ Softwares.

However, with pixelshader version 2 compatible graphic adapters, nearly all 2D effects can be processed smoother an faster (many pixels in parallel, hardware accelerated) and in higher resolutions on the GPU, and the result can directly be used as a texture in complex 3d scenes (or simple 3D effects), without additional memory transfer. Most developers of VJ Software have meanwhile realized this, and it is only a question of time until effects get recoded as pixelshaders and the software rendering pipelines get adapted to juggle with 3d surfaces, rendertargets and textures rather than system-memory-buffers. However, as long as there is only one single system-memory-buffer based effect in the whole software rendering pipeline, the bottleneck of expensive system-memory transfer still remains. It is important to note that this also applies for the "Video Decoding": there is no way to directly decode videostreams into GFX-memory for software based on OpenGL (where frames must be copied from system memory to GFX memory after decoding); this is currently only possible under DirectX (using DirectShow,VMR9 DS-Filter). Many programmers, Linux- and Mac-users are not aware of this, but it is no surprise that there is a difference between a platform independent architecture (OpenGL) and a platformspecific but more optimised architecture (DirectX). Because video decoding is such an essential part of VJ software, it must clearly be pointed out here that all VJ Softwares which use OpenGL pay this certain penatly in performace.
This does not mean that such VJ Softwares are performing bad, but it means that they will not scale in resolutions and layers (e.g. only 4 simultaneous layers of PAL video, rather than 8, and problems with HD video decoded simultaneously in multiple layers).

The speed of decoding videostreams itself mainly depends on the used codec. For VJing it is better to NOT use super-compressing codecs, because they will need more hardware resources for decoding. And the implementation of different codecs can vary quite a lot by different manufacturers too: only very few take advantage of hardware accelerated decoding and are optimised to be used with VMR9 (e.g. nVidias Pure VideoDecoder, which even works well on ATI cards :-)).

Effects
Effects, effects, effects... ...there can never be enough effects! Everybody likes them, everybody wants them. No doubts, there was a real effect-boost in the history of VJing when "Freeframe"
- an open-source cross-platform real-time video effects plugin system - was started by Pete Warden. Based on a simple C++ API, (system-memory based) Freeframe effects
can be hosted in any VJ Software that provides a corresponding wrapper (such as e.g. VisualJockey :-)). On the one hand, many interesting Freeframe packages are around, but on the other hand Freeframe exactly suffers by the performance problems mentioned above. There are discussions about "Freeframe 2" (aka Freeframe GPU), which will specify a crossplatform API that allows hardware accelerated effects using OpenGL. Unfortunately, this will again not result in ultimate performance, because of the mentioned video decoding problems related with OpenGL.

GUIs and Automation
Some VJ softwares such as e.g. VisualJockey do not provide predefined functionality concerning resource-, transitions-, effect-, and mixmode selection, nor do they force its users to use a predefined GUI for live shows or direct mapping of parameters. Instead, they just offer
the users a GUI with plenty of possibilities to design and arrange unique rendering-pipelines (in VisualJockey they are called FXCs) where all possibilities of resource/transition/effect/mixmode can be scripted and later remotecontrolled through whatever peripheral the user likes. On top of that, they offer functionality to define event-driven playlists, where sequences of events and corresponding actions (e.g. a smooth continuous rotation) can be triggered with input signals - typically MIDI. Although this requires additional understanding and more preparation time for a show, many advanced VJs would not want to live anymore without such powerful "internal processing and scripting concepts". And as a nice side-effect, such concepts also allow all kind of automation besides the normal "live VJing".

Good examples of such automation made with a customized version of Hippotizer (version 1)/VisualJockey (version 3.5) were realized by Luc Lavergne and Yan Breuleux: Ars Natura, Black Box. Another good example was 'Cyberhelvetia' at Expo02 in Switzerland, made with vvvv.

In any case - be it live VJing or automated setups - the beauty and professionalism of digital art has constantly grown, and VJing has become extremely popular. The nicest thing about VJing is that the resulting digital art is not at all predefined but truely realtime, always dependent on direct or indirect human interaction, and is thus the longer the more also understood and accepted as a new kind of art.

The following shows some pictures of one of my early VJ gigs (Energy 2002, Switzerland, about 16000+ attenders). The reason why I present this and not another event is because some key people from Green Hippo especially came to Switzerland to test the very first beta Version of Hippotizer V1, and it turned out to be mind-blowing!

Some (but not all!) control monitors from the VJ corner.

...and the start of the success story of the Hippotizer! ;-)

...the audience...

The hall was full of nice lighting effects all the time!

And this funny gentlement are nobody else than James Ross Heron and Sean Westgate - the founders of the company Green Hippo.

Oh and who's that guy on the left? It's Peter Kaufmann (aka pDoom) - the legendary coder of some stunning demos, the hacker of DM2, and meanwhile Lead Programmer of Green Hippo in London. The guy on the right is again James Ross Heron.
The next photo shows me and my friend Vadim Gorbatenko - the mysterious russian scientist working for Green Hippo somewhere deep in the snowy forests in Sibiria (Tomsk). I am still very proud that I got him to travel to Switzerland just for this event. It seems that he was a bit jet-lagged but also very amazed about the stunning output of the Hippotizers.

The guy in the middle (with the unblue T-shirt) is Kay Huber - my school day friend, who taught me programming gfx effects while attending the Geography courses at Gymnasium.

And finally, some snapshots of the big screen, showing some VJ output. The big screen was in fact so big, that it was impossible to fully get it into camera... :-)


James from Green Hippo also made a nice short videoclip!