Wednesday, February 18, 2015

We are drugs; speculative dev tools and psychedelic hologram futures.

This post is adapted from a talk I gave at Indiecade East 2015, where the theater was way too small for the crowd, so not many people got to see the talk... sorry / oh well. Here's basically what I said:

Our story begins on October 8th, 2014, on a very special episode of the Late Show with David Letterman. He was ending that episode with a musical guest from Japan -- a holographic vocaloid named Hatsune Miku. Pay attention to Letterman's barely-veiled incredulity as he introduces her. He can't believe the words coming out of his mouth:



But what really makes this moment is the ending, after the performance. Letterman doesn't even know what to say, and he knows he doesn't know what to say. The experience was completely overwhelming, so Letterman has to somehow pivot back to interpret it for his audience (mostly moms and dads from Milwaukee) and all he can muster is a facile comparison to "being on Willie Nelson's bus." (Willie Nelson, if you're not familiar, is a celebrity notorious for his drug use, among other things.)

The meaning is both clear and agreeable: Hatsune Miku is drugs.

Hatsune Miku is also partly a story of a Japanese indie developer named Yu Higuchi who made a tool called MikuMikuDance -- a freeware animation / choreography tool that can render out to AVI video files... countless music videos. Think something like Garry's Mod or Source Filmmaker, and then multiply the user base and cultural significance by a factor of 10, and that's Hatsune Miku: a pop sensation that sells out stadium shows while remaining accessible to anyone with a laptop and a YouTube account. Yes, Hatsune Miku is technically owned by a private media company called Crypton Future Media, and they definitely profit from her likeness, but she is more commonly understood as "indie culture" made possible by a free tool that leans on game development workflows. In a way, we are all Hatsune Miku!

... which means we are all also drugs.

"Modern Rhapsody" (1957), by Salvador Dali; earliest painted depiction of a "video game"
Writing in 1971, the legendary video artist Nam June Paik wrote that “[the] camera made everybody an artist", creating a new sense of participatory artistic culture. However, this artistic culture is also conflated with drugs, where "pot is a shortcut reaction […] to regain the sense […] of heightened participation." The surrealist painter Salvador Dali famously quipped, "I don't do drugs. I am drugs."

Both artists are known for being "trippy", yet from their perspective, they were just being artistic. Saying art is a metaphor for drugs is getting it the wrong way around -- rather, drugs are a metaphor for art. If we too become artists, then we too will be drugs, part of Paik's participatory culture.

Game culture has trouble reconciling the auteur-developer identity with the demands of industrial technocapitalism. Games are beautiful subtle works of art, except when We Demand Some Goddamn Service Here, and now artists are supposed to be factory managers. Imagine telling Salvador Dali he wasn't painting fast enough. (He would've laughed at you and told you to fuck off.) When this does happen (often) in art, we understand it as a craven and unethical practice. But in games, productivity-worship is the norm: Unity will enable the perfect realization of our intentions! Now with more control than ever before, 200% more innovation!

Our mistake, as video game artists, is assuming that this is the best way, or worse, that this is the only way. It is not. Sometimes you need a break from the relentless pressures of making a buck, you know? This is why we need speculative development tools, or "drugs" as I'd like to call them right now:

In Michael Brough and Andi McClure's drug "Become a Great Artist in Just 10 Seconds", drawing is willfully indirect and obtuse. The "lessons" playfully mock you for your inability to paint Starry Night, ironically emphasizing that "study" through repetition and similarity aren't the point. In fact, you'd be hard-pressed to repeat any single performance in Great Artist, because every key on your keyboard executes a different (secret) operation and irrevocably mutates the canvas, and the most compelling results emerge from timed combinations of keys. When in doubt, just do what resident Great Artist Liz Ryerson suggested to me: make ample use of [M] and [UpArrow]. What do those keys do? There's only one way to find out.

The drawing drug Alchemy, by Karl Willis, Jacob Hina, et al, seems disappointingly practical at first. You'll quickly recognize a toolbar and sliders for clearly labeled parameters at the top. However, you'll then discover that none of the tools are "practical" in the classical sense. Speed Shapes scribbles erratically according to your cursor velocity, Mic Shapes requires you to talk and shout to control brush size, and Ribbon Shapes seems tailor-made for making your own Joy Division t-shirts. Every tool distorts your cursor movements in somewhat unpredictable ways. Much like Great Artist, the computer is sharing the act of drawing with you, encouraging messiness and spontaneity. Maybe true participation means not wielding complete control over something?

Neil Thapen's 2D drug Doodal argues that this messiness can be ordered. Here, every drawing takes place in a separate layer / "frame", that can be freely moved and adjusted at any time to create recursive fractal structures. This is a very different workflow than Photoshop, where layer transform controls are hidden by default and force you to confirm any changes -- in Doodal, you move frames frequently, constantly rotating and sliding and scaling. What's stunning about Doodal is that you have more control over the drawing, but a bit less control over the composition; the emergent fractal patterns might be totally deterministic, but they will still surprise you.

Andi McClure's Icosa is sort of a surprise engine. (Andi is so cool that she appears twice in this list.) The first step is usually to generate a scene, but the text-only parameters are inscrutable (what does "face-aware" mean?), so your only real option is to press [R] to randomize the parameters and then jump in to see what happens. The mirror-in-a-mirror device totally works, but I feel that Icosa truly shines when you leave the outer shell and see the "hall of mirrors" generated from intentionally not clearing the frame buffer, so you get these beautiful cascades that transform the act of first-person-looking into an act of painting... well, painting with a rainbow-colored rhododendron that is imploding on itself.

Gabriel Gianordoli's Ribbons extends this "shape as painting" metaphor. First, start with some strokes, maybe even throw in some small bits and flakes. Then, go to one of the cloth sim modes in the top-left of the sidebar: repulsion, attraction, or wind -- and your drawing will come to life as an assemblage of physically-simulated ribbons, pushing and pulling and bending on each other. Ribbons imagines drawing as an act of collaboration with wind, with rhythm and movement. What if drawing was more like dancing?

I imagine Stephane Ginier's SculptGL as the successor to DrPetter's Sculptris, except SculptGL is free and open-source, and thus hopefully immune to a buy-out by the makers of ZBrush to shutdown active development. If you've never worked in 3D before, I highly recommend starting with a sculpting program like this one -- first generation modeling tools like Max or Maya use vertices / edges / faces as their form of 3D, requiring the user to manipulate these coordinates with mathematical precision. In contrast, to sculpt, you can just smash your mouse cursor in some virtual clay and see results instantly. It brings a softness and messiness to 3D modeling that wasn't there before. (NOTE: Currently, there's no robust texture painting or texture export in SculptGL, because I think there's no auto-unwrap thing like in Sculptris yet, but... it is open-source, so you can either help out or be patient.)

Structure Synth, headed by Mikael Hvidtfeldt Christensen, goes the opposite direction. Instead of sculpting visually, you sculpt textually by "coding" the 3D modeling algorithm itself. The language is simple and there are many learning examples included. What if modeling was less about working from representation or concept art, and more about tweaking numbers to direct the computer in modeling structures with you? (For its 2D predecessor, check-out Context Free Art.)

Terry Cavanagh's music box Bosca Ceoil (bus-ka kyo-al) is a compact MIDI music sequencing / composing drug. With only three modes at the top, it rejects the feature-bloat and distraction of other music tools, constraining everything to loops with specific keys and scales. The focus is refreshing -- not-musicians like me can just jump in and draw a smiley face and hear what a smiley face sounds like. Perhaps my favorite feature, though, is that enabling a new MIDI instrument will prompt Bosca to randomly initialize one for you. It's like the program is suggesting, "how about some nice woodwinds here?" And then you wonder, hmm, maybe woodwinds would help here...

Most of Phosfiend Studios' work in FRACT OSC was on the "campaign", a stunning first person musical adventure game. In a way, you could think of it as a very long lesson on principles of composing electronic music; each solved puzzle feeds into a fully-featured holographic music studio. At its core, the studio in FRACT is fundamentally similar to Bosca Ceoil and other step sequencer drugs... what's different is the spatial treatment, the thought to unify all these tools as a room with an ambiance and feel. When you turn up the "gravity" knob, whatever that does, it feels different from adjusting voice or pitch; the UI feels grounded in a way that Garage Band doesn't, helping you roleplay as the Daft-Punk-Cyber-Music-God you know you are. This is probably the first digital audio tool that would work well in a virtual reality environment.

Mark Wonnacott's experimental forks of SFXR, collectively called IECSFXR, are fascinating for the alternate workflows they imagine for sound synthesis. I'm only going to highlight my favorite, EVOSFXR, which uses an evolutionary model for "breeding" sounds. To use it, just select 2 or more sounds, make them have totally-reproductive sex, and then adjust the slider at the bottom to select certain children out of the possible generic variants. The simple one-line thumbnails are essential, allowing you to match certain visual traits to audio traits, an unusual way of visualizing sound other than waveforms or spectrum histograms... which is beside the point, because this is about sounds fucking! I think he should've called it "SEXR" or "FUCKXR" but hey it's not my project.

Imitone, with tech by Evan Balster and design by Dick Hogg, is a music composition drug. Instead of manipulating sheet music with a mouse or keyboard, you simply sing into a microphone, and the sophisticated pitch analysis thing converts it into MIDI signals for other programs to understand. What excites me about pitch composition tools is the idea of feeding-in non-human inputs -- record your car, record your dog, record the buzzing of a light fixture -- the world is singing, and now we know how to listen.

... and who could forget MikuMikuDance, or MMD. Because of its real-time nature, it borrows a lot from mainstream game development tools with notions of object hierarchies, keyframed poses, transform gizmos, and sorting orders. Unfortunately this is kind of a difficult drug to use... background AVIs must use the decades-old "Microsoft Video 1" codec, and the 3D model support is a mix of arcane .X files and their homemade .PMD character format, which is very jarring compared against the interoperability across most modern engines. It is its own walled garden, which makes it fascinating and a must-try, but ultimately kind of unpleasant. The salvia of toolkits.

* * *

All of these programs are useful for making "real" assets in "real" games. It's great if you end up doing that, but it's also not necessarily the point of speculative development. Much like speculative fiction or speculative architecture, the idea behind speculative dev is that feasibility and pragmatism can often limit our expression and ways of making. Here, the point is not the product, the point is the process -- the process of using these tools and feeling their grains and contours and suddenly wondering, "why can't Unity let me paint just by looking at stuff?" or "why can't I tell Maya to make these meshes fuck and have children and then selectively breed their offspring?" or "why can't I sculpt a gameplay mechanic?"

Too often in game development, we forget that this is supposed to be fun. To use these programs, I didn't take a class or buy a book. I just downloaded it and started using it. So many traditions of art-making like music, painting, or carpentry, all involve some sense of rejecting formalist intellectualizing in favor of just... "doing." Try to take joy in doing, collaborate with your computer, embrace messiness.

And if you do that, then the things you make will be intrinsically valuable, because they will be yours.



Thanks for listening.

This is an essay form of a talk I delivered at Indiecade East 2015 in New York City.