The Original Joytone

The first version of the Joytone was built as my senior design project at UPenn in collaboration with David Glanzman. The Joytone is a novel electronic musical instrument that uses a hexagonal grid of game-controller-style joysticks to make it easier to learn and play music with expressive synthesized sound.

Note: this project happened while I was in school, before my gender transition, so my old name is listed on most of these artifacts.


As a quick demo, here's a video of David Glanzman playing a Bach fugue on the Joytone:


My mom is a wonderful piano player and musical director - she even taught music classes for a while! Despite being around such amazing talent, I never learned to really play a musical instrument, though I had a vague familiarity with a bunch of music theory concepts. When I was in college, I made a bunch of musician friends and learned about all the quirks of their instruments of choice: how they could affect the sound, what scales were easy to play, how notes were organized. As a student with a burgeoning interest in user experience design, I became fascinated with the design of musical interfaces.

The summer before starting this project, I did an internship at DJTechTools, a blog / forum / shop for DJs who use software to play music. They also happened to manufacture some USB controllers of their own design. I spent that summer learning about, experimenting with, and critiquing the design of USB hardware for live DJ performance, and I carried a lot of that passion into my Senior Design Project for the UPenn Digital Media Design program. I decided to build a novel electronic musical instrument that tackled some of the limitations I saw in the design of acoustic instruments, and some of the limitations I saw in the design of the synthesizers available at the time.

I'm still working on building out this page in a pleasant way, but below I've put a link to the paper that I wrote about this project, which has a good level of detail.

In the mean time, here are some photo highlights!

The Problem

Learning music is hard. It takes years of practice to build up the muscle memory required to navigate musical structures like scales and chords. And as violin players know, it can be difficult to make notes that are pleasant, let alone expressive and emotional.

The Joytone was an experiment in musical interface design that focused on exploring those issues through the lens of an electronic musical instrument - a synthesizer.

Synthesizers get a bad rap: they're often criticized as being "cold" or "soulless". On top of that, the vast majority of them look like pianos, which only invites comparison to a grand, well-respected musical instrument.

I decided to look specifically at electronic musical interfaces because I saw two opportunities that seemed interesting:

  1. Because a synthesizer doesn't rely on acoustic sound phenomena (like the giant array of strings inside a piano), the interface is decoupled from the mechanism of sound production, meaning there are fewer constraints on the design
  2. With enough sensors, synthesizer parameters could be modified per-note instead of system wide, which could make it feel a lot more expressive and alive

And I had some goals: I wanted to build a new electronic musical instrument that was easier to learn than most "traditional" instruments, and more expressive than most synthesizers.


After doing some research about the design of musical instruments, I stumbled across a really interesting line of questioning. One of the reasons that learning to play the piano requires so much muscle memory is that it has white keys and black keys. Specifically, the fact that they have different sizes and shapes and are in different positions makes it hard to connect the structures you can hear with the structures you can see.

As an example, if you take a musical structure like a Major scale (shown below), and you play it starting from a C key, it's very easy! You just have to play the seven white keys to the right of the C key in order. However, if you want to play the same scale starting from a C# key, the physical hand movements you have to make are totally different! As a matter of fact, there are distinct hand positions for all twelve starting notes.


However, if the keys are all the same size and shape (if the keyboard were "isomorphic"), this problem evaporates!

Using isomorphic grids to simplify music theory isn't a new idea. In fact, Euler came up with the Tonnetz diagram as a way to explain the relationships between musical notes in 1739! The most familiar example is probably the grid of small keys seen on the left-hand side of an accordion (they make it easy to play chords under the melody typically played with the right hand).

But there are plenty of examples of this in electronic instruments as well. From left to right: the Rainboard, the C-Thru AXiS 49, and the Thummer.


I used an app called Musix Pro to experiment with different tilings, and for the Joytone I settled on the "Park Layout". The musical leaps between notes can be seen in the diagram below. What I liked about this layout is that all the in-key notes are concentrated in a thick band, making it easy to jam and stay in key! All the instruments about use a layout that optimizes for playing major and minor chords by organizing them into little triangles.


Zooming into the example we used earlier, we can see that the same musical structure now maps to the same physical structure (just pretend for now that the hex diagram on the right has a flat sign next to each note name - they got lost I think... oops! We're under construction...).


Experimenting with different ways to express relationships with color (screenshots from Musix Pro app cobbled together in photoshop)

Paper prototyping for hexagon size and the number of notes available

Our six-note prototype of the end-to-end system. It worked! And it was really fun to play with too!

Here's an overview of the system architecture. The joysticks are connected with common power and ground buses along diagonals. The signal wires (2 per joystick for the x and y dimensions) are connected through an array of multiplexers that pass along selected signals to the analog inputs of the Cypress PSoC 4.

The PSoC was internally configured to have an internal multiplexer routed to the analog-to-digital converter. The values read from the joysticks are passed along to the RaspberryPi in response to a python script, which then processes the signals, does some math, and sends out control signals using the OSC protocol, which is picked up by Pure Data and used as inputs for a synthesis patch.

We had planned to also implement some fancy light effects using an Arduino Micro connected via I2C, but we didn't get there in time.


And here are some build pics! The enclosure is a wooden box with a shelf that the acrylic mounting plates rest on, and the faceplate is screwed into the top. Attached to the bottom of the case is the RaspberryPi and the multiplexing board with the Cypress PSoC 4.


The Final Result

Here's a shot of the end product. It was amazingly fun to play with, and it wound up having a cool, drone-y organ sound to it. The continuous per-note expression control invites the player to explore constantly evolving tones, and the isomorphic keyboard makes it easy to explore musical structures.

After publishing this project online at my old website, we got some really cool press. The project was featured in MagPi, the RaspberryPi magazine, and we even did a couple of interviews!

If you want to read more... check out version 2, which was on display at a museum in Toronto!