TAOR continues the research tradition of eleVR, a group that existed from early 2014 to late 2017. But our research lineage goes back to the 1960s and includes many other names. These are my (Vi’s) personal recollections and how my views on art and research have developed during the past couple decades.
In 2013, Alan Kay approached me about a research group he was gathering with support from SAP, particularly Vishal Sikka who was the CTO there at the time. I myself was at Khan Academy, working on sharing genuine mathematical ways of thinking through video pieces. Alan undestood the research aspect of this work and asked: would I like to be a Principal Investigator in a new lab in San Francisco along with Bret Victor and Dan Ingalls? The new group was to be called CDG, the Communications Design Group. A purposefully vague name as to not overly constrain our scope. Yes, I could research whatever I want. Yes, I could hire a team.
Needless to say, I was all in. I joined in late 2013, and the group started coming together over the next six months. Bret found a beautiful office near South Park in SF and we set to work on making it home.
Alan and Vishal take a “People, not projects” approach to research, and showed great trust in hiring and funding us in a way that wasn’t attached to any particular project or outcome. I’ve since come to better understand their view on pure, long-horizon research. If you’re looking to invent entirely new ideas, you can’t define your research within the bounds of existing ideas. You have to choose people to trust, and then trust them. You have to support them even when you don’t quite understand what they’re doing.
As I spent more time around Alan and his cohort I came to appreciate the history of this kind of research and the ways in which our current group was modeled after his time at ARPA in the 1960s and then Xerox PARC in the 1970s. The development of computing came from time, funding, and trust in people. So they’re coming from a lived reality that research might take decades, and that while there’s a risk it won’t work out in the end there’s a real chance it could lead to a billion dollar industry. More importantly, there’s the chance that it could lift all of humanity and expand our ability to understand the world.
I also came to understand a certain perspective about thinking, particularly the human ability to understand new kinds of things and think in new ways. Computers may be an impressive physical technology, but they also come with a mental technology: computational thinking.
I had been used to thinking of technology as having to do with physical objects, from computers to farming equipment to the printing press. But I learned from Alan the value of including intangible skills when we think about and research about technology. Fire itself wasn’t the technology, but the knowledge of how to make and use it. Language was a purely mental technology for a long time, with literacy being a more recent invention to the species. A thousand years ago only the elite could read and write, now we expect every child to learn.
With the right tools and education, it’s not unreasonable to think that most any child could learn programming skills beyond any of our current abilities. After all, current programming languages are slow and clunky like writing in cuneiform, but someday (soon? maybe even invented by someone at CDG?) we’ll have the ballpoint pen.
Once I shifted my perspective to include mental technologies, I started seeing them everywhere. Skills, techniques, and cultural practices, in the history of science, and mathematics, and the arts, that are continually expanding what it means to be human. It’s a supremely optimistic view. Our brains haven’t changed much over the millennia, but our minds keep gaining new superpowers. It would be too much of a coincidence if we just happened to reach our limit now in 2019. What else is out there and how do we imagine it?
How did Ada Lovelace come up with a conceptual understanding of algorithmic thinking, and a vision for the potential power of computing machines, with only Babbage’s written plans for a giant mechanical calculator to work with? It was an impossible leap of imagination. After her work it took almost a hundred years, until Alan Turing, before these ideas reemerged. What if Turing’s work, as with Lovelace’s, didn’t find the support necessary to follow it up? How long would we have had to wait this time?
After a little time at CDG It became easy to not just imagine, but truly feel, the possibility that if things had gone just a little differently we would not have computers today. Government funding for research in the 1950s and 60s led to the computing revolution, the internet, VR and AI, and the more I learn about what went on back then the more I realize we’re still catching up. Despite how inexpensive, powerful, and ubiquitous computers have become, very few of the ideas floating around in technology today are better than the ideas from half a century ago, and quite a few are quite a bit worse.
Now back to 2013, when I wasn’t quite sure what I was going to do. My first vague notion had been that since Alan was interested in my video work, I could hire a team to produce more, better, videos representing true mathematical thought. But this didn’t quite feel right, as it would be addressing a current educational need rather than taking the kind of long view I was just starting to understand. Maybe I should work on inventing a whole new mathematical notation or mathematical thinking process. Maybe I should change directions entirely for a while.
In the mean time, we had this great space to host workshops, talks, and mathematics and art gatherings, and I worked on building community and culture.
When VR started making a comeback with consumer headsets on the horizon, I saw an opportunity to influence a developing technology before the direction became too set. After hearing heart-wrenching tales from the history of computing, of all the tiny decisions that no one thought were important at the time but that we’re stuck with today, VR was a chance for a fresh start. And as someone who had worked with video and was expecting VR video to be a thing, that seemed like a sensible place to start.
I asked Alan for hiring advice. He said: hire artists. Hire people who create compulsively, who cannot help but do what they do. Hire artists, and then trust them.
In retrospect I doubt he meant it literally, but I hired artists.
After all, I was an artist! Most people assume I have a math degree, but by the time I got to college I already had spent enough time in the field to know that what they teach in school captures very little of the beauty of mathematics, and I could find that elsewhere. More than I am a mathematician I am a composer, and my degree is in music.
I was briefly a professional composer before going over to tech. My first published videos were not of mathematics but of art music performances. I combined these passions into papers and talks on mathematics and music, and eventually adopted the title Mathemusician.
Here I should mention another branch of our research lineage, and another name: Reza Sarhengi, who was a mathematics professor, an artist, musician, and the founder and president of Bridges, a mathematics and art conference and community.
I went to Bridges for the first time when I was 13, and was immediately hooked! Reza welcomed and supported everyone into the mathematical art community, valuing all of their contributions, whether they were pure mathematicians printing clumsy computer renderings and calling it art, or pure artists using triangles as a metaphor and calling it math. At that age my perspective was very focused on the technical and I didn’t have much knowledge or respect for the “nonsense” spouted by artists. I was lucky to have met someone like Reza to show me how to be open to other people and perspectives.
Another transformational interdisciplinary conference community in my life has been the Gathering for Gardner conference, where incredible mathematicians and magicians collide, and it all makes sense somehow. Many such events were formational for me, but Bridges was the first community I was really a part of.
I returned every year I could, as an attendee, and then speaker, and then organizer. I learned an invaluable lesson from my years of experience as part of the organizing team, ignoring my instinctual academic judgements and doing things the Reza way. It wasn’t about making our conference look very impressive in one snapshot of time, but about creating a context where folks can immerse themselves in new ways of thinking. I saw myself, and others around me, grow from being wide-eyed beginners to being the experts and mentors who create the environment that lets the next group of beginners learn.
People like myself can grow because people like Reza allow them in. He was truly an incredible person.
Anyway! Alan Kay has a lot of thoughts on the culture of research labs, and about context and culture in education. A school or community with a music culture will make children into musicians. To make mathematicians, you need a math culture.
This made sense to me. I knew I only loved mathematics because I grew up in a mathematical household, playing with mathematical toys and with access to my father’s amazing books including Flatland, Martin Gardner’s columns, Raymond Smullyan’s logic puzzles, and many many others. If I had only my public school math class, I knew I would hate math like so many others who are cruelly conditioned to be afraid of one of life’s greatest pleasures.
And while our household coding culture wasn’t as strong as our math culture, it was at least there. I remember writing my first (very short) text adventure in QBasic as a child, and I took college-level computer engineering and computer science classes when I was in high school. These things had more of an influence on me than I realized at the time, because the mental superpower of algorithmic thinking doesn’t require a computer to work. I also started to learn a few things about programmer culture that weren’t particularly attractive, despite that I immensely enjoyed the material.
And suddenly ten years later my job was to help create a research culture, one where new researchers would naturally develop into people capable of thinking new thinks! A culture that is focused on the longer term, on process instead of on products, something more like Bridges culture and less like startup culture. But how?
By 2013, I’d had plenty of first hand experience doing pure research based on artistic motivations. I’d published on symmetry in music, computational origami, and computational balloon twisting. I’d given talks and workshops around the world, exhibited and performed in art galleries internationally, and did a short artist residency at Harvard. I’d made a video series called “Doodling in Math Class” that went viral on YouTube, leading to a profile in the New York Times. So when Alan said “hire artists” this made perfect sense to me!
So, I hired artists.
Andrea Hawksley is a mathematical artist who I’d met as part of the computational origami community, and also a skilled software developer. I knew she fit the bill of being a compulsive artist too: when Andrea is around, all nearby objects are in danger of being collectivized into a mathematical union, probably highly symmetric, possibly higher dimensional.
M Eifler was someone I’d met only recently, but every time I saw them they shed art as if it were a natural byproduct of human existence like an exhalation. They had some familiarity with virtual reality technology with their current job, and when we interviewed I learned that they also had a fine arts degree and previously had a job performing with a professional dance company.
The three of us got together and came up with a plan. I knew nothing about VR at the time, and for all my experience in theoretical computer science the practicalities of software development were beyond me, so M and Andrea told me what’s what and together we founded eleVR! All I had to do was trust them to be good at what they do, whether I understand it or not. And they are, as it turns out, incredibly good.
The early history of eleVR includes the eleVR player in May 2014, which was the first web player for VR video, and led to some press and industry connections for us. But the eleVR Player was art motivated—we had made a couple spherical video pieces including what was probably the first VR vlog, and we needed a player for them, so we (mostly Andrea) made one. I won’t go too into eleVR’s work as it’s very well documented at elevr.com.
When we first met Elijah Butterfield, he was showing a VR project involving poetry, on his hand-made cardboard headset. Now poetry was motivating VR research! He was in high school at the time and became our first intern, and later he returned as a contractor.
Evelyn Eastmond was working with a different group at CDG, and I didn’t spend much time with her at first. I knew she was a rockstar from the original Scratch development team, with an engineering degree from MIT. And she also has a fine arts degree from RISD, which led to her and M collaborating and talking art in a language I didn’t understand. I eventually paid enough attention to what they were doing to realize that ideas and techniques in the fine arts go far beyond what I was used to seeing with art-motivated technical research, or at conferences like Bridges.
My perspective changed when Evelyn shared a document about critique in art and how we could apply those techniques for CDG’s research. Up until then, I was thinking of the connection between art and research as something fuzzy and cultural, as if artists just happen to be good researchers due to some instinct for creativity, or that art gives us life and soul that helps to motivate interesting ideas. But I learned more from Evelyn and M’s fine arts experience, and began to understand that more of these techniques could be formalized for use in the context of technical research.I began to recognize there were aspects of my formal music training that I used in technical research as well.
We emptied out our little office and I watched in wonder and confusion as M turned it into a room-scale art piece that existed as speculative design for a theoretically virtual interface for organizing knowledge in space. What was happening? Something was happening here. I didn’t know what it was, but I knew I wanted more. “Always Trust M” became my new motto.
In 2016 eleVR, and much of CDG, moved to the new nonprofit Y Combinator Research with the generous support of Sam Altman, under the name of HARC: the Human Advancement Research Community. This was an amazing research environment for us and we learned just how valuable great support staff are. We were still adjacent to other amazing researchers we could cross-pollinate with, but had a little more of our own dedicated space which was tremendously helpful.
Another change at this point was that while I was officially the PI, that aspect was mostly about supporting my team who I’d long since realized were much smarter than me. With the restructuring our team insisted on having the same salary and we took a collaborative approach to our work, with different members being the lead on different projects. With all the organizational rearrangement it was also a good time for Evelyn to officially join eleVR, and I am ever grateful because her work with us really took our thinking to the next level.
By this time commercial VR and AR headsets had controllers, hand tracking, and better spacial tracking than ever. We decided to focus more on space, bodies, and how VR and AR might allow us to think in entirely new ways by allowing embodied knowledge and computational thinking to interact. We were all coding in webVR at that point, creating experiment after experiment at a furious pace.
For example, I became obsessed with the Venn diagram, invented in 1880 by John Venn as a diagrammatic tool for propositional logic. We created a virtual Venn diagram house, with overlapping rooms, drawers, and furniture. We created a Venn diagram museum of mathematics, with overlapping interactive exhibitions on geometry, topology and algebra. We created physical hands-on exhibits to go alongside the virtual ones. What simple thinking tools, akin to the Venn diagram, might be yet to be invented for 3D space for prior lack of 3D sketching paper?
What can you think in a room that you can’t on a page? What if that page, or room, is computational and lets us think out loud in space with all the subtle knowledge we keep in our bodies, perhaps coding in space with the same speed and fluency as I’m used to improvising on the piano, harking back to Alan’s concept of the “piano of computing”? The vision has been around for decades, but now the technology is useable enough to really experiment with the language.
How do you invent a new language for thinking? Our best guess is that, like any language, people develop it together through genuine attempts at communication, together in a shared context over years, and that it probably won’t make sense to outsiders until they spend some time immersed in it. We designed a spatial programming language in VR and pseudo-coded together in our virtual office in a language that could only compile in our own heads, and it was amazing.
Our research became more conceptual, more about developing technologies for thinking than about shipping easy-to-understand demos. And we changed our mission statement. We weren’t just researching computational media for new ways of thinking, but for new ways to think, see, and feel. We wanted to get at the things we don’t know how to think, and value kinds of knowledge we don’t yet know how to pin down.
In my experience, and in the experience of many mathematicians I’ve known, pure math research has more to do with feeling than you’ll see reflected in a math paper. First you have a feeling about something. You just know it’s true, or that it wants to be true, or that it must be connected to this or that. You’ll prove exactly why that is only in retrospect, and write it up as if you proceeded perfectly logically. And sometime’s you’re wrong. But right or wrong, how would we ever find out without first an un-proved feeling about what to explore? What is this feeling and how do we develop it?
I take inspiration from the scientific method, another relatively recent invention in human history. It is an incredibly powerful tool for answering well-defined questions. But how do you find those questions? How do you turn vague curiosities into a hypothesis?
Curiosity is not a technique. Curiosity is a feeling. Technical fields cannot help but at least pay lip service to curiosity, given its undeniable utility. At the same time, technical fields like to pretend feelings are irrelevant. I think we just haven’t developed techniques for them. The field of art, on the other hand, has spent a long time developing techniques for dealing with vague feelings and turning them into something more tangible. Art knows how to ask questions. Art is comfortable with things that haven’t been defined.
I don’t know what the human mind does when it creates breakthroughs in understanding or becomes hooked on a compelling curiosity, but I do know that currently defined techniques in technical research are not sufficient to produce these kinds of leaps. Why would we expect to have all the techniques when science and technology are so new?
Mathematicians were working on the parallel postulate for two thousand years before the leap to non-euclidean geometry was made. Once we had the concept of non-euclidean space, the scientific method could work on answering questions about the curvature of the universe itself. But how do you make that conceptual leap? Current research methods are great at incremental change, and incremental change has given us incredible progress. But what else are we missing?
I think it’s a good bet that new conceptual tools akin to the scientific method have yet to be discovered. I don’t know if art is the place to go looking for them, but I feel like it is, with the same kind of gut feeling I’ve learned to listen to in pure mathematics. These kind of gut feelings aren’t always right, but they’re the best indicator I know of for what directions are worth exploring.
We named 2017 “The Year of the Body” and focused on the kinds of knowledge we’d seen ignored by the tech industry thus far. We reimagined our work space such that we worked mostly from inside VR, whether to code, prototype, or think out loud. Meetings were all in VR where notes and diagrams were created in real time as we filled the space around us with our externalized thoughts. We learned new forms of communication. We brought these ideas into real physical space as well.
By mid-2017 I felt like we were really starting to get somewhere. We’d worked together long enough, doing our research with art techniques in mind, to start to develop a common language. I was starting to catch up to where Evelyn and M had already been even as they were racing ahead, and the more I learned from them the more I appreciated their art expertise and why it was important. We had a research culture. A context. It was endlessly exciting and a whole world of ideas was opening up!
At the same time, we were running out of funding. While we had many strong supporters in the industry, in 2016 and 2017 the climate was not very good for pure long-term research. The funding we’d had lined up fell through, and we disbanded by the end of the year.
I crowdfunded my work through 2018, with the trust and support of thousands of strangers online who I cannot thank enough. M joined Jaron Lanier at Microsoft doing research on machine learning legibility. I was heartbroken about losing eleVR and needed space from the industry, but I really missed the work and our team and at the end of 2018 I began to make enquiries into what was up at Jaron’s lab.
Jaron did foundational work in Virtual Reality, has a passion for mathematics, and is quite an accomplished musician, so it’s no surprise we get along. He has also worked with Alan Kay and has a familiarity with that kind of research culture. So in January 2019 I joined M and Jaron at Microsoft, and we started looking toward the future again.
With over a year of hindsight we could look at our previous work through new eyes and reform into a stronger version of our vision. We decided to let go of the focus of research on AR/VR, and make it just one more tool in our research toolbox. We also saw that the whole art-based research thing wasn’t just a feel-good pitch of the moment. We were really on to something. The research really had given us new ways to think, see, and feel.
And so while we’ll be doing research on many different topics, and in collaboration with others at Microsoft as well as elsewhere, the common thread for us will be the development of art-based research techniques for technical research. Maybe we’ll end up with a process that’s useful to others, maybe it will only ever make sense to us. Either way we can only develop our research process through doing actual research, so we’re still doing research in AR/VR programming language design, machine learning legibility, and most recently data dignity and data economics.
Thank you for following us on this journey. And especially thank you to the eleVR team and all I learned from them. Feel some feelings, think some thinks.
Vi Hart
Notes and References