CANVUS

Project Summary

CANVUS (Computationally Arranged Network of Visual-Auditory Uncontrolled Synthesis) is an interactive audiovisual installation built primarily in TouchDesigner.

The visual elements and audio react to the participants' movements in real time using skeleton tracking.

It is a group experience, designed to react to up to three participants at a time, allowing each to control different aspects of the experience.

The a2ru Exhibition

This iteration of CANVUS was created and showcased at the a2ru Emerging Creatives Summit 2026, a three-day event focusing on cross-disciplinary collaboration.

Our interpretation of the theme "Rewilding" was reflected in both our interaction design and the audiovisual content of the installation. The relationship between the participants and the installation was designed to invoke a sense of personal "Rewilding", encouraging you to move dynamically, exploring your body and the space around you.

At the same time, the audiovisual content represents a tension between the natural and the artificial. On the left, rigid, man-made, inorganic. On the right, fluid, organic, and natural.

We had an amazing team of five members who worked on the project:

  • Jake Farr - Programmer, Interaction Designer, and Networking Engineer
  • Alayt Issak - Physical Media Artist, Installation Designer
  • Gauri Menon - Physical Media Artist, Installation Designer
  • Evan O'Dell - Audio Engineer, Sound Designer
  • Nemo Rosas - Visual Artist, Projection Designer

Accomplishments

Interaction Design

This project is very special to me from an interaction design perspective, because the effect it had on the participants was so tangible: we didn't have to tell people to dance or move in a certain way. The installation naturally encouraged dynamic movement and exploration.

Multisensory Design

This project engages with visual, auditory, and kinesthetic senses to create a more immersive experience. The visuals and audio work together to create a cohesive space, and the participant's interaction with the piece gives them a sense of agency that makes it feel like they are really part of the piece. Overall, all of this created a really compelling experience that we were really excited to share.

Rapid Prototyping

This whole experience was created in just three days, by five people who had never met before, all from different backgrounds. We had to iterate quickly, test quickly, and make decisions quickly.

Interdisciplinary Collaboration

We also had to figure out where each of our unique skillsets could fit into a project: between us all, we had past experience with procedural visuals, sound design, computer science, philosophy, studio art, marketing, AI, and more. Our team had 3 undergrads, a Master's student, and a PhD candidate. Despite our different backgrounds, we all found a place on the team, and all contributed something special to the final product.

What I Learned

Networking with OSC

OSC (Open Sound Control) is a protocol for communication between different software and hardware. For this project, I learned how to use OSC for communication between TouchDesigner and Reaper.

Leadership

More than on any previous project, I felt a sense of responsibility and leadership in the creation of CANVUS. Each individual on the team had a specific role and area of expertise that made the project possible, but I was in charge of connecting all of the pieces together.

In a literal sense, I put together the final TouchDesigner network, connecting the skeleton tracking data with the visual and auditory elements. On a broader level, my knowledge of MediaPipe, TouchDesigner, and OSC meant that I was able to facilitate the integration between everyone's work, and I played a key role in ensuring each individual piece would work seamlessly together.

Installation Setup

Michigan State University provided us with the necessary space and equipment to set up a proper installation, not just a simple demo. This gave us all extremely valuable practical experience in installation design. Some of my key takeaways include the details of setting up different hardware, as well as being able to adapt based upon the space and equipment you have.

We had to find a way to position the camera that was optimal for skeleton tracking, and discovered that a wide, downward-facing angle was the best. We also had to figure out cable management and optimal placement for our stereo speaker setup. It's also important to ensure that the room is dark enough for the projection to be clear, but light enough that people can move safely.

We originally planned the installation to be a projection on four walls, using 5.1 surround sound. Due to the constraints of the space we had to shift to two walls, and stereo sound. This was disappointing, but it also helped us learn how to adapt quickly, and I'm proud of how well we all handled it.

Going forward, we would like to put up the project in a more confined space, to prevent observers from across the room from getting picked up by the skeleton tracking system. This space also had pretty bad acoustics, and in a future installation we would like to have 5.1 surround sound or even Dolby Atmos for the audio mix. It would also be great to have spotlights on the space where participants can move around, without them shining on the projection walls.

Reaper and Audio Effects

Reaper is the digital audio workstation (DAW) that we used to create the dynamic audio mix for our installation. While I was not in charge of designing/planning the audio mix or sourcing the sounds, I was responsible for connecting TouchDesigner to Reaper through OSC, and had to learn to control effects such as volume adjustment and delay on our audio tracks using those signals.

Technical Overview

A screenshot of TouchDesigner labeled with sections for Signal Processing, MediaPipe, and Visuals.

Our final TouchDesigner network, with the visual output and skeleton tracking preview.

Networking

I was in charge of connecting all of the hardware and software components for CANVUS, including:

  • Input: Webcam for Skeleton Tracking
  • Processing: Two laptops, separately running TouchDesigner and Reaper.
  • Output: Two projectors for visuals, two speakers for stereo sound.

The laptop running TouchDesigner had to be connected to the projectors and camera via cable, while the laptop running Reaper was connected to the speakers. To allow the two laptops to communicate, we used Open Sound Control (OSC) over WiFi.

The separation of tasks onto two laptops was both organizational and practical: it allowed for easier collaboration and troubleshooting as we could work on both devices at once, and it also made cable management easier while ensuring we had enough I/O connections.

Motion Tracking

We knew for this piece that we wanted to use skeleton tracking so that a participant's movements could control the piece, encouraging them to move and giving them agency within the space.

We settled on using Google MediaPipe to do the tracking, since it is relatively accurate, and easily integrated within TouchDesigner using an open-source plugin.

Although we don't want participants to consciously notice what parameters we use, this is what they are:

  • Left and Right Wrist Positions (Vertical & Horizontal)
  • Left and Right Wrist Velocity (Vertical & Horizontal)
  • Average of Left & Right Hip Position (Horizontal)

All of these parameters are tracked separately for up to 3 participants. The wrist positions control 12 different audio tracks (2 hands * 2 movement axes * 3 participants).

The left and right hip are averaged to create a proxy for the horizontal position of each participant within the space. This way, moving their whole body left and right can control the audio output. We used this to create a dichotomy where the left side of the space increases the volume of audio tracks that are more industrial and man-made, and the right side increases the volume of audio tracks that are more organic and natural.

The visuals are controlled by the average velocity of each participants' wrists, with each participant controlling one of the following visual effects:

  • Time Scale (animation speed)
  • Quantization (pixelation effect)
  • Hue Speed (speed of color changes)

Audio

A screenshot of Reaper featuring 18 audio tracks with waveforms.

Our final Reaper project, featuring 18 tracks split across 3 'characters'.

The musique concrète compositions featured in CANVUS were composed by Evan O'Dell, using field recordings captured from various environments across MSU's campus.

The music was composed and rendered in Logic, and then transferred to a Reaper project. Using Reaper allowed us more easily interface with OSC, since Logic isn't designed to take in arbitrary OSC connections from software such as TouchDesigner.

The general idea behind the composition is that each participant would correspond to a unique 'character' in the composition, with a unique style of sound. This means the project meaningfully changes as soon as a new participant is tracked, emphasizing when a participant joins or leaves, and giving them agency over the composition.

In terms of effects, we kept things pretty simple with just volume adjustment on most tracks, as well as delay on a few. Nevertheless, these simple effects cause big changes on the overall final sound and make the piece feel dynamic and reactive.

Visuals

Two people silhouetted against a pixelated RGB background projection, making dynamic poses.

The project switches between 5 different 'scenes' of visuals on a timer. These scenes were created procedurally within TouchDesigner by Nemo Rosas. At a high-level, the key visual effect used for these scenes involved using an edge TOP on Simplex noise, and putting that into several feedback loops.

One scene was created using physical media: Alayt Issak and Gauri Menon used a bricolage approach with the random art materials that a2ru provided, and created an abstract video recording. This recording was then put into our TouchDesigner project, where Nemo applied several visual effects.

As described in the Motion Tracking section, all of these scenes are then affected by the participants' movements.