Musician and computer scientist Adam Stark talks to us about his work on the musical mi.mu gloves and the relationship between technology and music
Adam is a musician and computer scientist with a PhD on digital signal processing and audio analysis in live musical performances and installations. In 2012, he started working with a small team, including Kelly Snook and Imogen Heap, developing a new piece of wearable technology to help breakdown the barriers between musicians and machines, and performers and audiences: mi.mu gloves. Loaded with advanced motion tracking electronics and algorithms, yet keeping a stripped back aesthetic, the team behind the glove have created something that is set to disrupt the traditional ways musicians interact with technology on stage. Adam talks to us about the struggles of large-scale manufacturing for wearable technology and how these gloves are designed to be hacked, played with and will change how people work with interactive technology.
Q: What problems do musicians face when trying to form with computers?
A: Over the past twenty years, musicians have started to use computers more and more during live performances. This is no surprise, because amazing synthesisers, sequencers and other music software have been developed allowing you to make new sounds, ‘interactive’ music applications and more. But - computers are inherently dull, and not at all performative. Someone sitting with a laptop on stage looks very much like someone checking their emails at home. Existing solutions to this tend to be MIDI controllers - often resembling a set of faders, knobs and dials. These help a little, but moving a fader still isn’t very exciting to do or to watch, and it only allows you to control one thing at a time.
Q: How do mi.mu gloves solve this problem?
Our gloves are looking to make the performance of music using computers both multidimensional for performers and visually exciting to watch from the perspective of the audience. Imagine panning a sound to the left by pointing to the left of the stage or making a sound more ‘spacey’ by just raising your arm.
Q: What’s your background and how did you find yourself with mi.mu?
A: I’m interested in the relationship between musicians and technology, and how this affects both performance and composition. I explored this during my PhD thesis at Queen Mary University of London. Much of my work involves working in situ with artists, developing interactive technologies for their performances and installations. I got involved with developing the gloves in 2012, after meeting Kelly Snook through Twitter, and began working with Imogen Heap and the rest of the team from then onwards. mi.mu was formed a year or so later after we had settled on trying to make the gloves available to a wider group of musicians.
Q: What’s your role at mi.mu?
A: My main role at mi.mu is in developing the software that allows you to connect the gloves to music. I want to help musicians to define their own relationship between their gloves and the music they want to make, and so the software is evolving into an open platform for connecting gesture to sound.
Q: What’s your favourite function of the gloves?
A: I think it has to be the ability of the gloves to recognise postures - i.e. to automatically be able to tell the difference between an ‘open hand’, ‘fist’ or ‘one finger point’, or any other posture. This was developed by my fellow mi.mu team member Tom Mitchell and really contributes to the sense of ‘magic’ you feel when wearing the gloves.
Q: How do you think they could influence the way we experience, make and perform music?
A: My hope is that they will change the way people make electronic music. Personally, I find lots of commercial music software quite restrictive. It tends to push you towards musical ideas based around its limitations - fixed tempo music, loop based music, ‘quantised’ music where everything is perfectly in time. There has been some amazing music made this way, and I’m definitely not turning my nose up at it. But how do we get a more natural and human interaction with electronic music? Why can’t we perform it with our hands? How do we encounter the idiosyncrasies we do when making music with acoustic instruments? I hope that the gloves will be part of the answer to these questions.
Q: Could you tell us about your live demo at SXSW?
A: It was great! From the mi.mu team we had myself, Imogen and Chagall van den Berg connected via video link and Kelly Snook in person in Austin at SXSW. I don’t think we’ve ever done something like that before with that many pairs of gloves connected. Imogen explained the story of the gloves and demonstrated her glove setup, Kelly talked a little about other people who have been using the gloves, I talked about the rationale behind our software and demonstrated the gloves controlling an augmented piano, and finally Chagall performed her song Sappho Song using her gloves.
Q: What’s been the biggest challenge you’ve faced with the development of the gloves?
A: I think it’s how difficult it has been to go from making one pair of gloves, which was a huge project in itself, to being able to get the gloves out to lots and lots of people, and to make them affordable too. We’re still on that journey and have a lot of work to do, and I’m learning new things every day. But seeing the enthusiasm for what we’re doing and not being able to just give out lots of pairs of gloves has definitely been the hardest part of the journey for me.
Q: The gloves clearly offer a practical and creative solution for musicians, but they look incredibly cool and are begging to be worn too. One of the biggest problems with wearable tech is designing something that people want to wear and won’t just sit in a draw after an initial play. Is the fact they look good a happy by-product of the practical needs?
A: This was definitely no accident. Rachel Freire, Hannah Perner-Wilson and others on our team have worked incredibly hard on making something that wasn’t just functional but also very beautiful. The gloves are intended to be used in performances, on stages, visible to lots of people. It is so important that they look the part, and as a performer, make you feel like you look good wearing them too.
Q: Could you tell us about your work with Imogen Heap?
A: Imogen began working with the idea of musical gloves from around 2010, and had a couple of initial exploratory performances with them. The first full glove song was the track ‘Me, The Machine’ from her latest album ‘Sparks’. The song was written entirely with the gloves in early 2012 for a concert that was live streamed from her garden. This was the project that initially got me involved with the gloves and we worked very intensely during that time to produce quite a sophisticated prototype that has heavily influenced the current software and hardware design. Imogen had a number of performances with this system including at TED Global, Wired and the AHRC.
In the video for ‘Me, The Machine’ Imogen linked her gloves to all kinds of different visual effects, so that what you see in the video is actually her controlling the visual projections with her hands.
Since then, she’s expanded her work with the gloves to explore a more improvisational approach, where she can easily create something from nothing, expanding it gesturally as she goes. As well as her existing setup, she has connected the gloves to Tim Exile’s system built in Reaktor, with help from Ash Dorey, which has added lots of new ways that the gloves can be used. This was visible in her improvised performance at Reverb festival last year at the Roundhouse.
Q: When do you think we’ll see a plug and play version of the gloves for the average human
A: This is absolutely what we are working towards, and it would be amazing if we can make it happen. In one sense we are already there - we have fifteen different people, we call them “collaborators”, around the world now with pairs of gloves, using them for all kinds of different things, from music for film to storytelling to music therapy. You can see a few examples in this video:
But here’s the tricky part; going from fifteen people to being able to produce thousands of pairs of gloves is probably the hardest thing we’ve ever tried to do. The technical, financial and organisational obstacles really make this very difficult. But stay tuned, we’re working on it!
Q: Has anyone hacked the gloves yet or found a different use for them?
A: Even better – we have spare input pins on our gloves so that other sensors can be connected to them. So in a way, they are designed to be extended by people. We hope that in time people might add extra electronics to measure the bend of their wrist or to connect extra buttons.
The gloves have been used for other applications too. For example, they have been used for controlling visualisations and projections and also for controlling small servo motors connected to Arduinos. There are definitely a lot of people who tell me “oh, your gloves will be even more useful in medicine/architecture/design/video games”. It would be amazing to try those things, and one day I’m sure we will take those steps, but at present our team is very focused on what these gloves can do for music as that is our passion and what we know best.
Q: Finally, what’s your favourite hack at the moment?
A: What comes to mind is a project led by my former colleague at Queen Mary, Andrew McPherson. Inspired by the way in which many innovations with music technology have come about through misuse of that technology, like overdriven amplifiers or scratching turntables, he started a project to design a ‘hackable’ musical instrument that invites the user to change the way it works by hacking it and altering its electronics. The result was the D-Box and you can see a video here:
Photo credit to Magda Wrzeszcz.