The Joytone, version 2
12 min. read
Hidden musical patterns
This project is a continuation of work done on the first Joytone - a new musical instrument that uses joysticks arranged in a hexagonal grid to allow the player to experiment with musical patterns and rich, expressive synthesized sound.
The original Joytone was built as a senior design project by myself and my project partner David Glanzman at University of Pennsylvania in 2014. This second iteration was redesigned and built for display at the Toronto International Film Festival's Kids DigiPlaySpace in 2015, which saw 16,000 visitors this year!
Building a new Joytone that needed to be engaging, reliable, and even more fun was a really interesting process, and that's the focus of this story.
(in the below video, the Joytone is visible at 6:55 - it's also listed under my previous name, which I legally changed this year)
To get a quick feel for how the Joytone looks and sounds, check out these two videos: a quick overview of the newest version of the Joytone, and a demo video of David Glanzman playing the original Joytone.
Learning music is hard. It takes years of practice to build up the muscle memory required to navigate musical structures like scales and chords. And as violin players know, it can be difficult to make notes that are pleasant, let alone expressive and emotional.
The Joytone was an experiment in musical interface design that focused on exploring those issues through the lens of an electronic musical instrument - a synthesizer.
Synthesizers get a bad rap: they're often criticized as being "cold" or "soulless". On top of that, the vast majority of them look like pianos, which only invites comparison to a grand, well-respected musical instrument.
I decided to look specifically at electronic musical interfaces because I saw two opportunities that seemed interesting:
- Because a synthesizer doesn't rely on acoustic sound phenomena (like the giant array of strings inside a piano), the interface is decoupled from the mechanism of sound production, meaning there are fewer constraints on the design
- With enough sensors, synthesizer parameters could be modified per-note instead of system wide, which could make it feel a lot more expressive and alive
And I had some goals: I wanted to build a new electronic musical instrument that was easier to learn than most "traditional" instruments, and more expressive than most synthesizers.
There are plenty of details about the design process of the first Joytone on this page, but the short version is:
- by arranging a set of identical sensors that play notes in a hexagonal grid, we created an interface where musical structures like a major chord have a consistent physical shape, making it easier to learn those structures (for comparison, a major chord is played differently on a piano depending on the note you start on, as a result of the white and black keys being in different positions)
- by using game-controller-style joysticks as the sensors, we were able to pack two dimensions of expression data into a familiar control that's easily operable with a single finger
- by creating a custom synthesis patch we were able to modulate aspects of the synthesized sound individually for each note, which allowed really interesting and unusual expressive play
We prototyped a bunch of options, we tested it, we hacked together a big final version, and it turned out to be a really compelling experience. It was built around a Raspberry Pi, and the project gained some popularity on maker blogs.
One of the organizers of the Kids DigiPlaySpace reached out to me and said he had seen the Joytone in an article and wanted to know if we could lend it to him for a show. He asked how many thousands of interactions it could withstand, and all I could think of was the hacky wiring job we'd done:
At this point I had moved to San Francisco and started working at VMware. I didn't have access to the shop resources or electrical engineering lab I enjoyed in school, so I got a membership at TechShop SF and got to work.
After building the first Joytone, we got plenty of feedback, and made a number of observations about how people played it.
- Users had a natural urge to explore repeating geometric shapes, and were delighted to hear repeating musical patterns as a result
- Users needed encouragement; they were worried about making an unpleasant sound (either just a harsh sound, or playing a note that was out of key)
- Users were nervous about damaging the instrument (the very DIY appearance of the instrument didn't help)
There were also a number of new constraints:
- It would be on display for six weeks, likely running all day, and the electronics would have to be much more reliable
- It would need to be serviceable by people at the museum, in case something fails (I would also need to supply backup parts)
- The hardware would need to survive ~15,000 visitors, most of them children
- It would need to look appealing, friendly, and approachable
- It had to be ready by February, leaving only four months for production
So I had my new goals: the Joytone was already engaging, but now it needed to be reliable and approachable.
The original Joytone consisted of a couple of high-level components
- Rows of joysticks wired together to share common power and ground wires but separate signal wires
- An "aggregator board" that collected all the signal wires and multiplexed them
- A Cypress PSoC 4 that did very high speed analog reads
- A RaspberryPi that got raw data from the PSoC and processed it into commands using the OSC protocol so it could talk to...
- Pure Data, an audio synthesis programming environment where I built a custom patch that had separate expression controls for each "voice" (the user could play 6 notes at a time)
This was a cool setup to experiment with and prototype out our vision, but it needed to be more reliable (and serviceable by museum staff who only have access to written instructions).
Custom Printed Circuit Boards
I decided to make custom circuit boards for the rows of joysticks, eliminating a huge amount of manual wiring work.
I started out drawing circuit board designs and reading up on tips and tricks for making custom PCBs. I had decided to slightly expand the keyboard to cover 9 identical rows of 8 joysticks each, for 72 total. The stepped shape of the original Joytone's keyboard was chosen as a balance between the kinds of notes available and minimizing the number of components we had to wire up. With custom PCBs, it was easier to just make all the rows identical, so I decided to go for more joysticks.
After running some ideas by an electrical engineer friend of mine, I started working in ExpressPCB to do the actual component layout
Even after checking my work over and over and over, I was still nervous to actually send out the design - I had never done this before! With some beginner's luck, everything came back exactly as I'd hoped, and I got to work assembling each of the 9 rows of joysticks that would make up this instrument.
Below you can see the interesting bits of this circuit: at the end there's a spot for an 8 pin jack that will connect to a cable going back to the Arduino. Right next to it is a mount point for an analog multiplexer - a kind of switch that would allow me to scan the signals coming back from each joystick by reading them one at a time in sequence. More details about the design of this circuit are below, in the section "The Brain".
I wanted the Joytone to look more polished, professional, and sturdy than its predecessor. I had also observed people struggling to see the playing surface and operate it comfortably with their hands. Thinking about the fact that my new users were children, who are generally short - I decided to give the case an angle to put the controls more closely at hand.
I also wanted the Joytone to be serviceable by the museum staff. I decided to include extra rows of joysticks and an extra Arduino pre-flashed and ready to go, and I spent a lot of time thinking about how to allow someone to easily and safely disassemble the instrument.
Below are some of my sketch thoughts.
The design I settled on is visible in the last sketch (and the photos below):
- The back panel of the instrument is just a snug press fit and can be pulled off like a lid
- The joysticks and LEDs are mounted on a thick sheet of acrylic that rests on a wooden rail and can be slid back, tilted, and lifted out
- The wires connecting the acrylic board to the arduino and other guts are long enough for someone to pull that assembly out and rest it on a table without unplugging everything
I decided to switch from the Cypress PSoC 4 + Raspberry Pi to an Arduino + Mac setup. The Raspberry Pi was fun to prototype with, but ultimately had too many reliability issues for me to trust it to just work for 6 weeks straight.
I had to redesign the aggregator board that connects everything together. As an extra challenge, I wanted to try to fit everything into the footprint of a standard Arduino shield - below are some of my circuit sketches.
I mentioned earlier that each of the 9 rows (with 8 joysticks on each) had a multiplexer at the end. The multiplexer has 16 analog inputs and one output. It has four channel select wires that determine which input gets forwarded to the output pin. So my strategy was to have the four wires representing the four bits of the channel number connected to all 9 multiplexers so I could control them all at once.
The Arduino would write 0000 to the channel select wires and read all 9 signals, then write 0001 to the channel select wires and read all 9 signals again. 144 analog reads later, I would know the state of the board and could process it into OSC commands to send over the main USB port. Fortunately, the Arduino Mega was fast enough to do this without the user noticing!
In the diagrams below you can see how the power, ground, chip enable, and 4 channel select wires were common between the headers for all 9 joystick boards, but the signal wires were separate (routed to A0 - A9 analog pins). This board also has connections for the LCD display, addressable LEDs, and function buttons.
Aside: The absurd number of header extenders in that fourth pic is so I have room to connect an in-system programmer to the USB chip because I was doing weird things with the USB firmware to make the Arduino show up as a MIDI device when you plug it into the computer.
Image credit: https://thefantasticgirl.wordpress.com/2015/03/21/creativity-centre/
One huge opportunity for visual appeal was the lighting system that didn't quite get finished in the original Joytone. The joysticks are surrounded by an acrylic honeycomb that separates the colored LEDs in each cell, allowing for some pretty interesting effects.
Below are some maps showing the basic note layout pattern colored in different ways (these are screenshots photoshopped together from an app called Musix Pro that allows you to experiment with note tiling patterns).
- The "Piano" scheme: notes that are "in-key" (in this case, in the key of C Major) are colored blue, with the starting note (C) colored green
- The "Rainbow" scheme: notes have the same colors across octaves (e.g. all Cs are red)
- The "Assistive" scheme: this was sort of a kooky idea, but as part of a music theory helper feature, it might even be possible to show users the appropriate finger positions for different music structures. In this example, the green pattern is a major chord, the red pattern is a minor chord, the orange pattern is a major scale, and the purple pattern is a minor scale - and this is true anywhere you play that pattern on the grid.
I wound up having time to build the first two, and created a third "neutral" mode that just left the grid in a white-toned state to avoid unnecessary distractions.
While I didn't manage to build any "play along" note highlighting guides or chord suggestion features into the light system, I did start to add some interactivity. Currently, active notes get colored blue, but I think there's tons of opportunity to give an abstract understanding of how you're affecting the sound (by modulating brightness, color, or patterns over time).
In this version of the Joytone, I also added a screen that explains what's going on with the lights. Specifically, it lists which notes are currently being played, the specific scale highlighted by the lights, and which coloring scheme is active. There's a great demo of this in the Joytone 2 Sneak Peek video at the top of this page.
Figuring out how to squeeze in all the necessary information on a 20 character, four line display was challenging, but I managed to get all the essentials in. One thing I struggled with was communicating the name of the selected color pattern: here, the lone word "Piano" sounds like that's the selected digital instrument! I believe I later changed this to "Scales" but I don't 100% recall.
The second Joytone came out really really well. It survived all six weeks of display without issue, and after going down to the UPS office to retrieve the largest package I'd ever received, I got it back in working condition. It did have some wear and tear, however. Specifically, I hadn't anticipated how dirty the clear joystick caps would get - they were pretty significantly discolored after so much play. The frost acrylic inset had also started to become unglued - in the future, I might find other ways to make these components last.
Shortly after getting the Joytone back, I disassembled it because I wanted some of the parts for another project. I still have all the components however, and plan to use them to explore other musical interfaces (or for quick demos of the concept!).
I think there are a lot of interesting directions to take this work. I would love to investigate ways to not have to rely on Pure Data for audio synthesis - it could be an exciting project to build custom synthesis systems into the instrument and focus on giving it a unique, recognizable sound rather than trying to make a general-purpose controller. Another aspect of music interfaces that I find really interesting is the way that musicians can play them without looking. The two-dimensional grid of the Joytone means that there are fewer easily recognizable spatial cues for "where you are" in the musical web of notes, and that makes it harder to play without looking.
So... who knows! Maybe there will soon be a third member of the Joytone family...