Masking Machine is a wearable AR apparatus and performance which brings audience, artist, and algorithm face to face. It was inspired by a photography series of the same name.
Masking Machine, documented here performed at the YBCA Bay Area Now 8 opening.
“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks” Published in ICCV 2017
Collage is the language of the moment, but has been for over 100 years. Lets walk through where it came from (Dada), what it’s up to now (Instagram), and why it’s integral to the future of AI (Deep Fakes, GANS, and the ingrained copy).
Hannah Höch, Da-Dandy, 1919
While the technique of cut and paste in its simplest form dates back to the invention of paper in China around 200 BC, collage didn’t become an indispensable mode of communication until mass-produced, printed media was adopted as a material by artists in the early 1900's during the very early stages of Modernism. Enter, stage left Hannah Höch.
Höch was a German Dadaist working in the inter-war Wiemar period and on through WWII. Her works poked surreal at the failings of the Wiemar government, her era’s social constrictions and prescribed roles for women, and an ever increasing flood of pop culture. In a fractured and churned up time, Höch constructed her works from fragments of fashion magazines, newspapers, and photographs stating that there were “no limits to the materials available for pictorial collages — above all they can be found in photography, but also in writing and printed matter, even in waste products.” (1) Her work embraced contradictions, and kneaded together the gender, racial, political, and material tensions of the time including jumping headlong into the art vs. craft tribalism that is still pervasive today.
“[Embroidery] is an art and ought to be treated like one … you, craftswomen, modern women, who feel that your spirit is in your work, who are determined to lay claim to your rights (economic and moral), who believe your feet are firmly planted in reality, at least Y-O-U should know that your embroidery work is a documentation of your own era.”
— Hannah Höch, in Embroidery and Lace, 1918
But Höch, and this is one of the reasons I chose to start with her, didn’t just shred this trove of printed materials, she also produced them. For over a decade, as Madeleine Boucher of Artsy.net writes, “By day, Höch designed embroidery patterns for women’s hobby periodicals and illustrated daily magazines, while by night she quite literally ripped those images apart in her artwork. Her work often violently attacks the ideals of femininity and glorified domesticity that those magazines repackaged and popularized as fashionable.” (2)
Collage stakes value in this kind of dual relationship to the omnipresent materials of culture but this stance is double edged. David Banash, author of Collage Culture, argues that the use of ready-made components was central to art making in the 20th century because while it highlights “the possibilities and limits of an inescapable consumer culture, “ and “has the power to be critical of [its] ideology,” collage also “depends on the materials of consumer culture and quite literally re-enacts the exact processes of mass production and consumption.” (3)
Hannah Höch, Untitled (Large hand over woman’s head), 1930
The collage aesthetic in all its material forms not only helped to spark Modernism, it made huge gains in cultural relevance through Postmodernism and beyond. Let’s take a quick side trip thru that history so we are all on the same page about these concepts and periods.
Eras are a bit like stereotypical teenagers rejecting their parents’ ways. Modernism had the Enlightenment to fight against. The Enlightenment, which was from about 1620 to 1800, mostly describes early scientific and anti-monarchy activities happening in Europe. Modernism, which began in the late 1800’s, rejected the Enlightenment’s certainty and reductionism. Postmodernism, which emerged after WWII, cast off Modernism’s objectivity and universality. And today, in a period I like to call the Transist era, we are sloughing off Postmodernism’s irony and anything goes attitude.
“In considering the names that might possibly be used to designate the new era following “postmodernism,” one finds that the prefix “trans” stands out in a special way. The last third of the 20th century developed under the sign of “post,” which signaled the demise of such concepts of modernity as “truth” and “objectivity,”…”originality”… and “sincerity”… All of these concepts are now being reborn in the form of “trans-subjectivity,”… “trans-originality,” “trans-lyricism,” “trans-sentimentality” etc.”
Mikhail Epstein, Russian literary critic, The Place of Postmodernism in Postmodernity (4)
And through all that upheaval, collage, with its devotion to hybridization, liminality, and intersectionality, proved an important and durable tool for art and communication. “OK, OK, so collage is great and useful and has this whole lineage or whatever. What’s your point? You promised memes and AI!”
Instagram Story post by the author
So while the producer-consumer hybrid was born in the cauldron of photo collage, it has spent the last 100 years chewing its way into every corner of social, political, and artistic life. Even the process of writing this piece is enabled by the collage aesthetic. I’m clipping quotes from online resources, reorganizing my words on the fly, and pasting the product into a publishing platform. Think hip-hop and sampling, think Pride and Prejudice and Zombies, think inspirational speech supercuts, think every meme you’ve ever participated in, think social media platform interface design and development.
If this were a podcast, this is where I would insert the record scratch.
Because it’s not just the images we use to advertise our avatar-selves on Instagram, or even the functionalities and interfaces provided in the app that have taken on the cut-and-paste aesthetic, but the techniques used to make the the software itself. Instagram started out as a fairly simple photo feed but didn’t catch proverbial fire until it began directly copy-pasting features from its competitor Snapchat. Disappearing photos and videos sent to individuals or groups of friends, which later became public ‘Stories,’ face filters, and decoration with animated stickers all originated with Snapchat. (5)
This is not necessarily a value judgement against Instagram’s design practices, (I have plenty of beef with their choice to include intentionally habit-forming functionality, but not with its copying of competitors) but instead is meant to point out that collage is integral at all scales of techno-communication today. And it’s not just big platforms constructing themselves from bits and pieces taken from elsewhere. Software development these days never starts from blank canvas. There’s nearly always a library, API, or component system snagged from zee interonets or internal legacy code. And sure, yeah, maybe I’m stretching the concept of collage well beyond the confines of its original meaning, but it turns out language is made by those who use it. Get off my smoke-smothered sidewalk language police! (The Camp Fire still burns as I write this. Today the AQI in San Francisco is 265.)
Like evolution, which cannot jump from a present body format directly to the perfect ideal but instead must step blindly through a thicket of tiny mutations, technological change tends to improve incrementally as well. But then, we invented this thing called genetic engineering which allowed huge changes in genetic makeup that would not have been possible via evolution alone. Collage is the genetic engineering of technological innovation. Much the way working in diverse groups of people recombines their varied backgrounds and ways of thinking into more wide-ranging possible outcomes, collage expands the boundaries of what’s possible to create by bringing into contact material of various sources and types
Art-based research for technological exploration, the style of research I specialize in and have done at SAP, Y-Combinator Research, and now Microsoft, leans heavily on collage as an inquiry technique. Rather than focus on incremental improvement, art-based research attempts to jump to far off, precarious, and often not-viable-at-present lily pads in order to broaden our menu of innovative avenues. Collage is a skill, an art, and the key to democratizing the future of AI.
Image Source: NVIDIA via NYT
Laura Hoptman, former curator at the Museum of Modern Art in New York, says collage allows us to “to experience information simultaneously… as a horizontal cloud.”(6) Seems an apt description for today’s AI: tools which essentially allow us to grasp our ever increasing sea of information through condensation and representation. AI is collage at scale. It’s predictions and generations are collections of human knowledge, as in labeled sets of cancer screenings, and material culture, as in troves of images, minced into vast arrays of values, scrutinized and recombined into new knowledge and material.
Image Source: NVIDIA via NYT
And as AI becomes more commonplace it will become akin to those hobby periodicals and illustrated daily magazines of Höch’s era, like the gifs and memes of today, itself subject to material recombination. This is already happening. Take generative adversarial networks, introduced in 2016 by Ian Goodfellow (7), a Google Brain researcher. GAN’s are a collage of two preexisting kinds of algorithms. It’s a technique used for creating new versions from a set of originals, for example creating an artificial voice from a library of human speech sounds or a new face from a collection of faces. Think of a GAN as a very patient artist and a very discerning buyer. The artist is commissioned to make a portrait and given a large library of source material. Every time the artist finishes a portrait the buyer gives them a score. At first it’s a pretty bad score, but soon, after say a million tries, the score gets much better (not perfect, but pretty close, hair and irises are still pretty tough).
Image Source: NVIDIA via NYT
Now because we are talking about AI we have to keep in mind that neither the “artist” nor the “buyer” has any idea what a face or even what an image is. Both break pixel colors up into arrays of values which they then run statistical comparisons on and make adjustments to. An AI, at least by current definition, can’t understand the problem set to it. It can only, very quickly, aggregate thousands of “data-fied” human actions, like labeling an image, or create statistically similar examples of a collection it is provided, like creating an image. That cancer screening AI might get it right more often than any individual doctor but that’s not because it’s “smarter” but because it uses the knowledge of thousands of doctors all collaged together. (Am I oversimplifying how these piles of calculus work? Yes. There are at least 1 trillion articles online which can teach you the basics. Go forth!)
Now Goodfellow is a super-extra-double expert and at the moment that’s what it takes to make a good AI collage. But what if instead of focusing all our effort on creating easier ways for people to code their own versions of established AI (generators, classifiers, anomaly detectors etc.), instead we facilitate wide ranging AI collage. Glue! Scissors! Tools galore! To empower a large number of people, everyone, bakery owners, digital artists, middle schoolers, to craft useful and meaning-making AI, we shouldn’t exclusively focus on ways that make coding novel models more approachable. The lineage of collage shows us that interfaces that give access to existing AI algorithms and let people play around with recombining them in odd and counter-intuitive ways will lead to an explosion in who is creating AI and what fun, weird and maybe even revolutionary things it’s being asked to do. This is how we avoid obfuscation via automation. This is how we deep-fake the meme wars. This is how we win!
Hannah Höch, “On Collage,” in Hannah Höch, ed. Dawn Ades, Daniel F. Herrmann (London: Whitechapel Gallery, 2014)
Madeleine Boucher, “Art or Craft?: Hannah Höch’s Collages Embraced the Conflict Between Art and Craft, Dada and Commercialism”, Artsy.net, published 10/14/14, access 11/12/18, https://www.artsy.net/article/madeleineb-art-or-craft-hannah-hochs-collages-embraced
David Banash interviewed by Rick Poynor, “Collage Culture: Nostalgia and Critique,” Design Observer, published 11/11/2013, accessed 11/17/18 https://designobserver.com/feature/collage-culture-nostalgia-and-critique/38187
Mikhail Epstein, “The Place of Postmodernism in Postmodernity,” published 1998, accessed on 11/17/2018, http://www.focusing.org/apm_papers/epstein.html
Karissa Bell, “Instagram is proof shameless copying pays off,” Mashable, published 5/12/18, accessed on 11/17/2018 https://mashable.com/2018/06/12/instagram-proves-copying-is-the-best-strategy/#NysItTmXfaqo
Rachel Wolff, “Cut-and-Paste Culture: The New Collage,” Artnews.com, published 12/12/13, accessed 11/18/2018, http://www.artnews.com/2013/12/12/the-new-collage/
Wikipedia, accessed 11/19/18 https://en.wikipedia.org/wiki/Generative_adversarial_network
Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, “Progressive Growing Of Gans For Improved Quality, Stability, And Variation,” Published at ICLR 2018, https://arxiv.org/pdf/1710.10196.pdf
The Masking Machine is a still color digital photography series using machine learning to create masks of the artists face. Each image begins life as a simple selfie, the most quotidian, maligned, and rampant photography of our time. Next the images are feed into a landmark detection algorithm which finds the position of my facial features. Using this data the artist applied a layer of digital makeup, contact lenses, or glasses. Once saved the images are fed back into the landmark detection algorithm and the process begins again, over and over, building up eerie distorted masks of the artists face.
Whether working in a dark room, coaxing chemical systems like wet plate collodion or cyanotype, or working with a black box, bartering with the algorithms like landmark detection or style transfer, what is photography if not scientifically processed image-making through a machine? And when done recursively, feeding the same image through that process again and again the machine itself also becomes visible in the images. Like repeatedly taking a pinhole photo of a pinhole photo until nothing remains but the artifacts of the photographic process itself made visible. Our time is caked with filtered images, enlarged eyes, and whitened teeth, all enabled by nearly invisible photographic algorithms. If we are to live with them we also need to recognize and understand them and Masking Machine is here to make them seen.
Update: Masking Machine was printed and debuted at the 2020 show Recoding CripTech at SOMArts.
Making the Bed is a VR library of a diverse set of beds, from royal to homeless. Each bed can be explored and lied in at both miniature and life-size.
Below is the text of a 2017 research write up of the project for EleVR, a reseach and Development Team at SAP
Making the Bed is a VR space developed in Anyland. When it first loads you are plopped down on a red rug surrounded by a cloud of miniature beds. Each bed, in this frozen avalanche of doll furniture is numbered, 1 – 50, and some have a little green dot over the number. That dot is there to indicate which beds you can visit and experience at full size. As stated in the instructions it’s best experienced seated on the floor with a pillow near by so that you can lay down in the full sized versions. Touch a number with the model of your out stretched pointer finger, number 18 say, and you are teleported to a life sized cot in a jail cell. Touch the number again to return home. Touch number 24 and you are teleported to a nest of pillows on a fire escape. Touch the number again to return home.
This exploration was sparked by a recent Year of the Body related discussion we had in which Evelyn mentioned an art school course she took called the Drawing Marathon which helped her shift drawing from a mentally directed activity (one of planning and judging) to a physically directed activity (one of arms and fingers making decisions). The class asked her to draw for 100 hours: 10 hours a day, 5 days a week, for 2 weeks. The assignments changed frequently: Draw for 30 mins, now erase everything you drew. Fill larger and larger surfaces. Draw for 30 mins, now tear your drawing into tiny pieces and glue it onto another sheet. Draw the same thing, a piece of fruit for example, 50 times.
That last direction, drawing the same thing 50 times, is the one I used for Making the Bed. The process was pretty simple. The first bed was just a pillow, a platform, and four legs. The next one had drawers underneath, so fancy. Then came round beds and four posters and cribs and bunk beds and by 10 I had completely run out of bed ideas. The idea of beds expanded to places that we sleep, like homeless encampments and fire escapes and jail cells, and places other animals sleep too, like nests and burrows. One bed just referenced migraines, something that happens frequently in my bed. But then I was out of ideas again and I turned to historical research to keep me going. Turns out the first beds were just piles of wool and straw but it wasn’t until the invention of the platform off the ground that beds benefits (less bugs and rats) really kicked in.
Then I went off an a metaphors and fairytales tangent making beds for the stories of Princess and the Pea and Goldilocks and the Three Bears as well as the sayings “Sleeps with the fishes” and “Dirt nap”. And eventually, in the home stretch, I realized I could make the beds any size I wanted and started designing the beds to be interesting to lay down in at life size. The layout of the final space: the mini map, the teleportation, the detailed larger versions, and the secret teleportation objects, all came out of that late stage realization.
That late epiphany about scale (a topic VR seems always to loop around back too) reveals one of the clearest advantages of the medium.
I started this project standing but at some point over the 30-40 hours it took to complete I took to sitting on the floor legs crossed meditation style or stretched out in a Y. It saved energy while still making it easy to maneuver around the space by sliding on my butt. I found my arms could work for long hours in what I have taken to calling the ‘hug range’: that empty space between your arms when you pretend to hug a large beach ball. I would on occasion reach out beyond the hug range to grab or place something but keeping my shoulder blades down my back with my elbows close in to my ribs let me work for longer.
(Side note: I have noticed that I prefer not wearing over the ear headphones while doing this work because the social hum of the lab makes the headset feel less isolating. I’m sure Chaim and Robert and Dan, my neighbors at HARC, can attest to the fact that I look ridiculous at best sprawled on the floor next to them all day with a giant black mask strapped to my head, flailing at the air with two black donuts on sticks, all while wearing an adult onesie I made myself.)
But satisfying my need to work in the hug range (a delightfully human need) also means that, at least initially, every bed had to fit within that small physical range. I made less detailed overviews as sketches. Then I scaled up the sketches and worked on them at a new level of detail. The difference here is that the same motor schema, hold up wait a minute:
Motor schema are a idea I am getting from George Lakoff and Mark Johnson’s, 1999, Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought. (Hit up page 77 for a more through introduction) A motor schema is a set of movements by which I interact with a basic level category of object like a car. When the motor schema for the use of two objects varies beyond a certain threshold we put the objects in different categories: car and boat for example. But if their motor schema maintains a general similarity (steering wheel, analogue acceleration etc) we group them under an umbrella term: vehicle. Motor schema variation is one way the body creates knowledge, how it categorizes.
So again the difference here is that the same motor schema I used create the miniatures, was used to create detail on the life sized versions as well. My body is thus categorizing those two levels of zoom as the same (or at least similar) tools. You might be used to that kind of motor schema flexibility in say Maya where zoom lets us use similar finger motions to manipulate at different scales but those are symbolic, abstract interactions via the interface of the keyboard and mouse. It feels completely different to have motor schema flexibility for the distance my actual hand is moving through actual space.
Making the Bed is open to explore via Anyland just go to areas and search for “MAKING THE BED”
Big Important Emotion, a 2 sculpture pair, are a pair of brown paper sculptures. Evolving from 2017’s series Unscannables, 3D paper collage sculptures exploring the line between what is possible in physical objects and surfaces and what can be seen and represented virtually, the Big Important Emotion diptych erupted at a moment of high stress and vulnerability in the artists life. All color and pattern striped were away They were made of peeled paper board, with all color and pattern striped away, and took on all the raw sharp defense so prevalent in moments of upheaval. This piece is now in the High Sierra Hermitage Collection.
Brain damage acquired in childhood has left M with minimal long-term memory. This project was their first attempt at building a prosthetic replacement. Using a combination of videos and writing addressed to their future self the artist spent May 2018 - August 2020 building up a dataset which would later be used in building Prosthetic Memory.
Invisible Sculpture is a performance structure for audiences to interact with an invisible sculpture via an AR headset. One at a time during the performance audience members get a chance to see the invisible and their act of looking will reveal the size and position of the sculpture to those around them. Originally designed to enable the artist to show monumental scale sculpture as guerilla art at SFMOMA later the piece was shown at The Wattis, UC Irvine, and TED.
Ken Becker and the artist sat down to talk about the piece in December 2016.
This piece focuses on the how’s of using art based research to produce better user experiences and technology in general. To read the full piece visit medium.
During their time at This Will Take Time they focused on drawing with an augmented reality headset. AR is an extremely ephemeral material as the drawings themselves could not be saved so only image and video documentation of the drawings survive.
The drawings include room scale abstractions, 3d figure drawings in situ within the furniture of the residency house, and performances in which the artist interacts with the finished pieces. They explore not only the drawings relationships with their surrounding but also how the artist’s body shares that space with them.
In Climb into Bed a gender ambiguous figure is arranged in a pose reminiscent from 19th century female nude paintings, but they aren't the only nude in the video. The artist climbs into bed with the figure drawing, attempting to touch their immaterial form, conjuring up ideas about consent, avatars, and touch across the digital divide.
Due to the limitations of the AR headset the artist also created a set of 5 speculative drawing performances for video to explore what could be possible with better hardware and software.
“I’ve not seen a piece like this before. Mesmerising.” -Viewer
Room Scale Abstraction
