Creative Projects

Oculizer

Over the past couple of years, I've been developing a DMX lighting automation system that creates real-time, music-reactive lighting. Oculizer uses machine learning to automatically predict and switch between lighting scenes based on live audio analysis. The system leverages EfficientAT, a state-of-the-art audio tagging neural network, combined with spectral audio features to understand both the semantic content and acoustic properties of music. This allows Oculizer to intelligently match musical moments to appropriate lighting scenes while using mel-scaled FFT to analyze frequency components and map them to DMX values through configurable scenes.

My hope is that this project is the precursor to developing systems that facilitate fruitful social interactions. Our brains are highly sensitive to the spectra of ambient light, sound, and odor in the environment, but the ways in which these factors impact our feelings and behavior often evade conscious awareness. Through a systematic research program of testing the relationships between these factors and human behavior, it's possible to build the knowledge necessary for us to optimize the spaces we inhabit towards connection, cooperation, and a really fun time.

The project is open source and available on GitHub.

Core Features

• Intelligent scene prediction using EfficientAT neural network embeddings
• Real-time audio reactivity using mel-scaled FFT analysis
• Dual-stream and single-stream audio modes for flexible setups
• Support for RGB lights, dimmers, strobes, and lasers
• Automatic scene switching based on audio content analysis
• Manual scene control with interactive grid-based browser
• Live scene switching and MIDI control support

Funded by an arts integration grant, I am working with my colleagues in the SCRAP Lab (Lindsey Tepfer and Mark Thornton) to build an augmented social experience that will soon be featured at the Hopkins Center for the Arts. Initial data collection is ongoing!

Our installation, titled "The Art of Conversation", is programmed social environment that stacks time on top of itself. Pairs of participants view gallery art together while having the semantic content of their conversations analyzed in real-time. The words they speak are used to index utterances from all previous conversations that were similar in meaning and played back to them in real time. This creates the effect where each spoken sentence trigger similar sentences from other conversations. Imagine describing how a piece of art makes you feel, and then hearing other people describe the same piece of art in similar ways immediately after finishing your sentence.

Artistically, our goal was to create a space where participants could see themselves in the context of the larger collective conversation - to reveal the forest to the trees and the commonality that their perceptions have with others. Scientifically, we are using this data to understand how people's conversations are shaped by those in the periphery. This project is the first of many that we hope will lead to systems that can adaptively intervene in social interactions while maintaining naturalism.

Much of my creativity revolves around hosting parties (for Halloween in particular) and making them as immersive and fun as possible. In the few months leading up to Halloween 2024, I managed to accumulate an array of cathode ray tube (CRT) televisions, a VHS camera, a couple of tiny hidden cameras, and an FM modulator that can send a wireless video transmission.

I used these tools to immerse my guests into multi-angle analog home video being generated in real time. When participants entered the house, they were confronted with a large stack of CRT televisions displaying video streams from around the house as well as from the VHS camera, which was completely wirelss and mobile. This allowed for guests to wander around the party with the camera, having their perspective visible to everyone in the room. In addition, the hidden cameras were camouflaged in Teddy bears and other objects around the house.

My favorite part was zooming the camera in on a single person's face as they arrive, creating the experience of arriving at a party and immediately seeing themselves plastered across many TVs. This created a unique social amplification effect, wherein one individual's emotions were disproportionately contagious to everyone in the room.