Open Source

October 18, 2013

Firstly, our apologies for not blogging more. We’ve been kept rather busy, and haven’t put in the time to update this blog that we should have. There are several projects that we’ll write about in due course.

First though, an announcement.

We’re pleased to announce the immediate availability of several libraries that we use to help power apps and websites. They are as follows:

All of these libraries are open sourced with a BSD license, so you are completely free to use them as you see fit.

We’ll post more about these libraries in the coming days and weeks.

We recently worked on the iTunes LP for Emeli Sandé’s “Our Version of Events” album, and great news, it’s number 1 in the UK album charts. Congratulations to Emeli and Virgin Records.

Pink Floyd iTunes LPs

February 6, 2012

We’ve just finished developing iTunes LPs for almost all of the Pink Floyd studio album catalogue. Three of the albums – Dark Side of The Moon, Wish You Were Here, The Wall – were given special treatment, and a series of visuals developed for them. They’re available on the iTunes Music store here.

All of the deluxe LPs feature Memorabilia sections, which build upon the work done for Papa vs Pretty, allowing you to zoom in and out of assorted Pink Floyd historical items.

Each deluxe LP also feature ‘Marbles’ – as each physical equivalent box set product came with a custom designed marble, the challenge was to incorporate this into the digital LP somehow. The Marbles visual is an area of optical distortion that warps the underlying image, so it looks as if marbles are rolling across the screen. it’s all done in canvas (webGL isn’t an option in LPs, yet), the challenge was to make it run well, so it’s highly optimised in what it does. We’ll try and write a more detailed post about this optical distortion at a later date.

Wish You Were Here featured a water ripples effect, again done using canvas.

The Wall is our first public use of the Three.js library, which powered the 3D brick wall constructed for use on the menus.

Papa Vs Pretty

February 6, 2012

We were asked to develop an iTunes LP for Australian band Papa Vs Pretty. The singer (Tom) had assembled a large scale collection of drawings, and sketches, that told a whole story along an irregular arc. The hope was to somehow work this massive image into the 1280x720px fixed dimensions of the iTunes LP. The original image was 20,000px across by about 10,000px high. Instead of cropping out selected areas and using those as backgrounds, we instead developed something to traverse across the whole image as a seamless whole, overlaying song/context specific information depending on current location. similar, in some limited respects to Google/Bing Maps. There’s a video of it in action here – they’ve dissolved between songs to give more of a taste to the album, but the LP itself traverses without any such fades. Hats off to Tom, and the EMI AU team for wanting to do something so interesting and different.

a while back (feels ages ago now), we developed the deadmau5 iTunes LP. There’s a blog post about it earlier. Thrilled to see it used an an example use case for the Google derived Web Audio API. Web Audio, and Mozilla’s audioData are two specs we’ve been experimenting with, and hopefully we’ll have something to show soon.

We’re especially proud of this LP, as it was a deliberate effort to try and imagine a different interface. Most LPs follow a standard pattern, with a top level menu, leading off to different sections – but we felt it was important to try and differentiate the types of LP designs we could offer offered, so that ultimately, different types of release could be achieved.

Plus Minus (+-) is a singles box-set in the physical land, and in the LP we’ve maintained this. Each single in the LP can have their own content/feature set, which leads to a lot of flexibility – you can explore each single as its own contained world, with accompanying essay, video, art and arbitrary types and amounts of whatever content needed or desired.

The initial research for this approach of LP interface design began with the idea of presenting an artists’ work through a horizontal timeline. Early on it was decided that if we were representing an artists’ work, then their events might be more than recorded audio, so the LP is indifferent to what the event is on the timeline – it could be a photoshoot, a single, an album, a video, a documentary, or <insert something here> :-)  The timeline becomes the top menu (homeController) with each event becoming its own controller, and each eventController can define its own content.

So, this blog article is admittedly a little late, but we’ve been kept busy. I’ve finally not got the time to sit down and write a bit.

Back in the middle of October 2010 we started work on the iTunes LP for Deadmau5’s new album 4×4=12. The most important and challenging part of the brief was that the label wanted a visualiser that would respond to the audio, and be in keeping with the Deadmau5 image.

By this time we’d written several different visualisers, making use of various technologies. Since viz development takes time and money, the most common option chosen by our customers is usually a simple slideshow. We do like to stretch ourselves though, so these often get augmented with additional bits of fun. For example, whilst Kylie’s “Aphrodite” has a slideshow viz it is framed by animated waves above and below the photos, and track titles that get animated onto and off the screen with a zooming effect when the track changes. The LP for Michael Franti & Spearhead’s “The Sound of Sunshine” has quotes from Michael Franti appearing to accompany the slideshow. Gatecrasher Anthems’s slideshow uses a fancy sharding effect to transition between photos. Iron Maiden’s “The Final Frontier” features four different visualisers, one that shows titles over a spinning starfield, the second is slideshow of some of the artwork created for the album showing their mascot Eddie and other aliens, third a colour-shifting photo of the band, and finally a 3D ‘skull tunnel’ viz inspired by the 80’s arcade game Tempest. (For good measure, the front page of Iron Maiden also features blobs of blood floating around space.) On a related vein, we have done timed lyrics displays for several LPs for singles, like Roll Deep’s “Good Times” and Professor Green & Lily Allen’s “Just Be Good To Green”.

There’s a few visualisers we’ve developed that for various reasons have not seen the light of day too. We had a viz that showed short fragments of lyrics showing up timed to the music, fading in and out as the songs are sung. Another that didn’t make it out was a fish-tank viz that featured several fish swimming about the screen, bubbles floating up, and lyrics that floated up the screen in time to the music.

Technically there’s a lot of different technologies going on in that lot. Many of them make use of dynamically generated HTML. We make extensive use of CSS3 animations, transitions and transformations, including 3D transforms – indeed the Iron Maiden ‘skull tunnel’ uses nothing but those techniques. The waves in Kylie’s LP are produced by JavaScript code using HTML5 canvas. That code monitors how long the system takes to draw those waves to ensure the wave animation is as high resolution and smooth as they can be – run that LP on a low powered PC like a netbook and the waves will be smoothly animated in low resolution, but run it on the latest and greatest and you’ll see full 720p resolution waves running at 60 frames per second.

Whilst we had done plenty of things timed to happen at specific times in tracks one thing our LPs hadn’t featured before starting Deadmau5 was a viz that directly responded to audio. We’d produced a few experiments for our own benefit that drew spectrum graph displays but they weren’t ready for prime-time; they looked a bit naff and didn’t really work very well. But now we had some time, some budget, so it was time to brush off that code, and get engrossed in some serious research and development. So I got to work.

I soon discovered a few tricks for dealing with the waveform spectrum data that one can extract from iTunes. The key to processing the data to make it look better is to group together values into separate channels. iTunes provides 256 useful level values over a linear spread of frequencies, however that’s not how we perceive audio. We hear sounds instead on an exponential scale; a middle C tone has a frequency of approximately 261.5Hz, C an octave higher is twice that at 523Hz, and C an octave higher still is 1046Hz. To produce a spectrum graph with 6 bars/channels for example, the 255 data values from iTunes’ data need to be grouped into blocks of 1, 2, 6, 18, 55, and 174 values. Armed with that piece of knowledge I set to work producing some prototypes.

Before long I had some simplistic spectrum bar graph displays running that, whilst quite crudely styled, gave a response that looked right. A steady stream of prototypes ensued, and little later they were looking like the LED displays popularised on stereos in the 80s, complete with decaying peaks.

Now prototypes are useful, but they’re not an end-product. We needed to come up with something that would look “Deadmau5″. If you look up Deadmau5 on YouTube you’ll find several live performance videos of him. There’s a distinct Deadmau5 visual style – he stands (wearing a large Deadmau5 head) behind a tilted cube-shaped DJ booth, where the faces of the cube are display screens, and there’s generally also a large display screen behind him. Various different animations and effects get displayed on these screens. These videos gave us plenty of ideas, something to aim towards.

Since the cube-shaped booth is an important part of the Deadmau5 stage performance experience we had to include that into our viz. Technically the cube is three HTML5 canvases rotated in 3D space using 3D CSS transforms to form the three visible faces of the cube. We had two parts of the puzzle – we knew how to do viz things that responded to audio, and we could do a cube display.

What was clear at this point is that in order to create an experience that looked like a Deadmau5 stage performance we were going to need a sophisticated viz engine. It was going to need to be timeline driven, and for that we could leverage off our previous experience with timed lyrics displays and timed commentary. The engine was going to need to support rendering to multiple different canvases, enable and disable different displays, and support several different types of animation with different display options, including of course spectrum bar displays.

I’ll go into some more specifics in a future blog post, but the high level overview is this:
The viz engine for Deadmau5 features a timeline management system coupled with a programmable rendering pipeline system.

The timeline system allows for actions within the engine to be called specified certain times during track playback. Each track has a separate timeline defined specifically for that track, and the timeline being followed will change when the track changes.

Rendering pipelines are defined at run-time. A pipeline comprises of multiple stages, each of which specifies a renderer along with a destination canvas and optionally also a source canvas. Additionally pipelines can be defined to call actions before and/or after the rendering stages are completed. Pipelines are usually started and stopped by actions called by the timeline system.

The Deadmau5 viz engine includes eleven different renderers. The simplest of these is the ‘clear’ renderer, which will clear a canvas, or fill it entirely with a specified colour. The most complex renderer is ‘vu’, which will generate various different VU displays. There are also a couple of effects-type renderers in the form of ‘dotty’ and ‘pixelate’, which allow for various different pixelation effects.

As well as pipelines, the engine provides a ‘settings’ system allowing for settings objects to be created that contain values that will persist and can be re-used between different rendering pipelines or actions. As well as adjusting and clearing settings, there are actions in the engine to increment values, or randomize them.

Finally there is a ‘transition’ system in the engine. This allows for settings values to be changed over a specified period of time as playback progresses. As well as transitioning numbers from one value to another, the transition system also supports transitioning colours, and includes a callback mechanism to allow a set of actions to be called when a transition has completed.

The engine design was modular to allow for things to be combined in most ways we can think of. Even with just 11 renderers there’s a very large number of possible displays the Deadmau5 viz engine can produce. We by no means exhausted those possibilities in Deadmau5…

In terms of raw statistics, the viz engine in the Deadmau5 LP weighs in at approximately 1700 lines of code, plus a further 2800 lines of data definitions for pipelines, settings, and timelines.

In my next blog post I’ll write in a bit more detail about the engine’s pipeline system and some of the renderers included in the Deadmau5 viz.

Development of the engine also hasn’t stopped with the Deadmau5 LP. The viz in the LP for Hillsong United’s “Aftermath” album makes use of a later version of same engine, albeit with less renderers (but including one new one) and a more sophisticated transition system. I’ll also be writing on how the engine has subsequently developed, and giving some insights into where it’s going.

this LP came with some really distinctive art direction, especially with regards the typesetting. Inside the CD equivalent, the lyric sheets are printed on vellum, which we replicated visually for the lyrics inside the LP.

The art direction frequently used subtle distortions, as if the art was viewed through a fractured lens, and this was a trope we referenced in the photo gallery: each image is split into shards, which then align themselves to reveal the original.

With such quirky and lively art direction, it was hard not to add in some animations for this LP. The home menu begins with two planes flying in and the island that Eliza is climbing over popping view. The kind of animations that make you smile when you’re doing them.

This LP was done for Virgin Records UK. It was a lot of fun, and had a great photoshoot and art direction underpinning it. We tried had to keep the interface of the LP as simple and out of the way as possible, to really let all the imagery have the space it needs to make impact. So, on the homepage, no album title, just Kate Moss’ face and a small menu tucked away.

Follow

Get every new post delivered to your Inbox.