Re-imagining the opera of the future

In the mid-1980s, composer Tod Machover came across a copy of Philip K. Dick’s science fiction novel “VALIS” in a Parisian bookstore. Based on a mystical vision Dick called his “pink light experience,” “VALIS” was an acronym for “vast active living intelligence system.” The metaphysical novel would become the basis for Machover’s opera of the same name, which first premiered at the Pompidou Center in 1987, and was recently re-staged at MIT for a new generation.

At the time, Machover was in his 20s and the director of musical research at the renowned French Institute IRCAM, a hotbed of the avant-garde known for its pioneering research in music technology. The Pompidou, Machover says, had given him carte blanche to create a new piece for its 10th anniversary. So, throughout the summer and fall, the composer had gone about constructing an elaborate theater inside the center’s cavernous entrance hall, installing speakers and hundreds of video monitors.

Creating the first computer opera

Machover, who is now Muriel R. Cooper Professor of Music and Media and director of the MIT Media Lab’s Opera of the Future research group, had originally wanted to use IRCAM founder Pierre Boulez’s Ensemble Intercontemporain, but was turned down when he asked to rehearse with them for a full two months. “Like a rock band,” he says. “I went back and thought, ‘Well, what’s the smallest number of players that can make and generate the richness and layered complexity of music that I was thinking about?’”

He decided his orchestra would consist of only two musicians: a keyboardist and a percussionist. With tools like personal computers, MIDI, and the DX7 newly available, the possibilities of digital sound and intelligent interaction were beginning to expand. Soon, Machover took a position as a founding faculty member of MIT’s Media Lab, shuttling back and forth between Cambridge, Massachusetts, and Paris. “That’s when we invented hyperinstruments,” says Machover. The hyperinstruments, developed at the Media Lab in collaboration with Machover’s very first graduate student RA Joe Chung, allowed the musician to control a much fuller range of sound. At the time, he says, “no serious composers were using real-time computer instruments for concert music.”

Word spread at IRCAM that Machover’s opera was, to say the least, unusual. Over the course of December 1987, “VALIS” opened to packed houses in Paris, eliciting both cheers and groans of horror. “It was really controversial,” Machover says, “It really stirred people up. It was like, ‘Wow, we’ve never heard anything like this. It has melody and harmonies and driving rhythms in a way that new music isn’t supposed to.’” “VALIS” existed somewhere between an orchestra and a rock band, the purely acoustic dissolving into the electric as the opera progressed. In today’s era of the remix, audiences might be accustomed to a mélange of musical styles, but then this hybrid approach was new. Machover — who trained as a cellist in addition to playing bass in rock bands — has always borrowed freely from high and low, classical and rock, human and synthetic, acoustic and hi-tech, combining parts to create new wholes.

The story of Dick’s philosophical novel is itself a study of fragments, of the divided self, as the main character, Phil, confronts his fictional double, Horselover Fat, while entering on a hallucinatory spiritual quest after the suicide of a friend. At the time of Dick’s writing, the term artificial intelligence had yet to achieve widespread use. And yet, in “VALIS,” he combines ideas about AI and mysticism to explore questions of existence. In Dick’s vision, “VALIS” was the grand unifying theory that connected a vast array of seemingly disparate ideas. “For him, that’s what God was: this complex technological system,” Machover says, “His big question was: Is it possible for technology to be the answer? Is it possible for anything to be the answer, or am I just lost? He was looking for what could possibly reconnect him to the world and reconnect the parts of his personality, and envisioned a technology to do that.”

A performance for the contemporary era

A full production of “VALIS” hasn’t been mounted in over 30 years, but it’s a fitting moment to re-stage the opera as Dick’s original vision of the living artificial intelligence system — as well as hopes for its promise and fears for its pitfalls — seems increasingly prophetic. The new performance was developed at MIT over the course of the last few years with funding from the MIT Center for Art, Science and Technology, among other sources. Performed at MIT Theater Building W97, the production stars baritone Davóne Tines and mezzo-soprano Anaïs Reno. Joining them also were vocalists Timur Bekbosunov, David Cushing, Maggie Finnegan, Rose Hegele, and Kristin Young, as well as pianist/keyboardist Julia Carey and multi-percussionist Maria Finkelmeier. New AI-enhanced technologies, created and performed by Max Addae, Emil Droga, Nina Masuelli, Manaswi Mishra, and Ana Schon, were developed in the MIT Media Lab’s Opera of the Future group, which Machover directs.

At MIT, Machover collaborated with theater director Jay Scheib, Class of 1949 Professor of Music and Theater Arts, whose augmented reality theater productions have long probed the confused border between the simulacra and the real. “We took camera feeds of live action, process the signal and then project it back, like a strange film, on a variety of surfaces, both TV- and screen-like but also diaphonous and translucent,” says Scheib, “It’s lots and lots of images accumulating at a really high speed, and a mix of choreography and styles of film acting, operatic acting.” Against an innovative set designed by Oana Botez, lighting by Yuki Link, and media by Peter A. Torpey PhD ’13, actors played multiple characters as time splinters and refracts. “Reality is constantly shifting,” says Scheib.

As the opera sped toward the hallucinatory finale, becoming progressively disorienting, a computer music composer named Mini appeared, originally played by Machover, conjuring the angelic hologram Sophia who delivers Phil/Fat to a state of wholeness. In the opera’s libretto, Mini is described as “sculpting sound” instead of simply playing the keyboard, “setting off musical structures with the flick of his hand — he seemed to be playing the orchestra of the future.” Machover composed Mini’s section beforehand in the original production, but the contemporary performance used a custom-built AI model, fed with Machover’s own compositions, to create new music in real time. “It’s not an instrument, exactly. It’s a living system that gets explored during the performance,” says Machover, “It’s like a system that Mini might actually have built.”

As they were developing the project this past spring, the Opera of the Future group wrestled with the question: How would Mini “perform” the system? “Because this is live, this is real, we wanted it to feel fresh and new, and not just be someone waving hands in the air,” says Machover. One day, Nina Masuelli ’23, who had recently completed her undergraduate degree at MIT, brought a large clear plastic jar into the lab. The group experimented with applying sensors to the jar, and then connected it to the AI system. As Mini manipulates the jar, the machine’s music responds in turn. “It’s incredibly magical,” says Machover. “It’s this new kind of object that allows a living system to be explored and to form right in front of you. It’s different every time, and every time it makes me smile with delight as something unexpected is revealed.”

As the performance neared, and Machover watched Masuelli continue to sculpt sound with the hollow jug, a string of Christmas lights coiled inside, something occurred to him: “Why don’t you be Mini?”

In some ways, in the age of ChatGPT and DALL-E, Mini’s exchange with the AI system is symbolic of humanity’s larger dance with machine intelligence, as we experiment with ways to exist and create alongside it: an ongoing venture that will eventually be for the next generation to explore. Writing thousands of sprawling pages in what he called his “exegesis,” Philip K. Dick spent the rest of his life after his “pink light experience” trying to make sense of a universe “transformed by information.” Though the many questions raised by “VALIS” — Is technology the answer? — might never be fully explained, says Machover, “you can feel them through music.”

Audiences apparently felt the same way. As one reviewer wrote, “'VALIS' is an operatic tour-de-force.” The three shows were filled to capacity, with long waiting lists, and response was wildly enthusiastic.

“It has been deeply gratifying to see that “VALIS” has captured the imagination of a new group of creative collaborators and astonishing performers, of brilliant student inventors and artists, and of the public, wonderfully diverse in age and background,” says Machover, “This is partially due to the visionary nature of Philip K. Dick’s novel (much of which is even more relevant today than when the book and opera first appeared). I hope it also reflects something of the musical vitality and richness of the score, which feels as fresh to me as when I composed it over 35 years ago. I am truly delighted that “VALIS” is back, and hope very much that it is here to stay!”