Once upon a time in Winchester, VA, a nurse and a psychologist wondered what to name their second child (a newborn boy). This little boy would one day grow up to be a famous politician, so it was important to give him a good name. Eventually they settled on Richard (which means “powerful leader”) for a first name, and John (which means “God is gracious”) for a middle name; they gave him his father’s last name, because that was the custom at the time. Yet today, when someone says “Santorum,” do you first think of the former U.S. Senator? Or do you maybe think of columnist and gay rights activist Dan Savage?

Much to the former Senator’s likely chagrin, “santorum” is an excellent example of how words (and objects) that were not originally intended or designed to be “political” can take on new meanings–as well as new politics–once out in the world. Certainly Aldo and Catherine (who created both Rick Santorum and his name) didn’t intend for “Santorum” to take on the meaning that it did in 2003; they gave Richard John that last name in order to mark his patrilineal heritage, and a future struggle over gay rights was probably the furthest thing from their minds as they did so. Their original intentions, however, couldn’t stop the name “Santorum” from taking on new meanings and associations once Rick had brought it into the public sphere. This process–the introduction of words and other signs into new contexts, where they take on different meanings–is often called “resignification.”

The history of the word “santorum” is interesting not just because of the word’s salacious new meaning, but because the change in meaning was purposefully enacted. Resignification is usually not intentional; rather, it arises slowly and subtly from shifts in a word’s usage as social conditions change. For example, because feminists have criticized the use of masculine pronouns to represent all humans, and because there is now broader social recognition of transgender persons, the use of “they” and “their” as singular third person pronouns has seen increasing acceptance (this use even has a name: it is “the epicene they”).

Similarly, words and ideas that (at one time) were not thought of as political can wind up at the center of major political controversies. As philosopher of science Bruno Latour notes, our tendency to perceive “nature” and “society” as separate and distinct–even though nature has never been separate from culture, and science has never been free of politics–makes vocabulary words from the natural sciences and from engineering particularly prone to sudden and unexpected resignification. Designers (for instance) may think that their work has nothing to do with politics, but this is never the case. We should therefore not be surprised (and neither should they) when terms that they coin for specific scientific or industrial purposes take on larger, more politicized meanings.

In this post, we’re going to look at how the single reality vocabulary–specifically, terms like  “mediated reality,” “mixed reality,” “dual reality,” “blended reality,” and “augmented reality”–came to describe not just specific digital technologies (as the designers who coined those terms originally intended), but also the general conditions of social interaction within our present-day world. We term this shift toward a broader and more politicized set of meanings the sociological turn, and it has been foundational  to much of the discussion on Cyborgology.

The Single Reality Vocabulary: A Reaction to “Virtual Reality”
As the Web was first emerging, observers and user interface designers alike found it practically necessary to conceptualize reality as something comprised of interactions between both atoms and bits. Without such a conceptualization, it would have been impossible to capture the hybrid nature of the kinds of objects and environments they were trying to create. The ontological shift came with little fanfare; there was no real discussion of its philosophical magnitude, nor of the sociological consequences that might follow from this new way of thinking. The designers simply coined new terms and proceeded with the business of building technology. Such terms included “mediated reality” (Naimark 1991), “mixed reality” (Milgram and Kishino 1994), “augmented reality” (Drascic & Milgram 1996; Azuma 1997), “blended reality” (Ressler et al. 2001; Johnson 2002 ), and later “dual reality” (Lifton and Paradiso 2009). Despite their humble origins in design studios, these single reality terms are early conceptual models of the relationships between physical matter and digital information. In other words, they are practical ontologies.

The single reality vocabulary is clearly meant to counter common assumptions that the physical (alternatively, “the real”) and “the virtual” are discrete, even causally-isolated, worlds. This idea–that atoms and bits are not only relegated to two separate worlds, but that these worlds can’t affect each other–is what we at Cyborgology call “strong digital dualism”; as we described last week, such dualism was fostered both by the cyberpunk imagery of the 1980s and by the technoutopian idealism of Silicon Valley in the early 1990s. Dualist assumptions about the Web are also only the most recent examples in a long history of both popular culture and academic theory associating electronic media with parallel or virtual worlds, as Jeffrey Sconce illustrates in his book Haunted Media (2000). The world described by single reality terms therefore stands in stark contrast to digital dualist imaginaries.

Digital dualists’ rhetorical emphasis on physical matter as constitutive of the “real world” clearly implies that the Web’s supposed virtuality is somehow not real. (It should be noted that calling the Web “virtual reality” is also an expansion of meaning; the term “virtual reality” was not coined to describe the Internet, as Adriana de Souza e Silva and Daniel Sutko [2011] demonstrate.) Dualists’ “real world” vs. “virtual reality” distinction stems from a hyperfixation on the fact (“fact”) that these two supposed worlds are comprised of two different substances (aka, atom or bits); similarly, dualists tend to ignore the fact that (for instance) both atoms and bits can act as media of communication between agents, actors, or consciousnesses. To a strong dualist, “the virtual” is something we enter into only by leaving behind the physical. (Think Neo jacking himself into the eponymous Matrix of that film, or Cobb and Arthur entering the dream world in Inception. Is this what happens when you log into Facebook? …yeah, we can’t get that feature to work, either.)

If we assume for a moment (as, regrettably, many others have over the last two decades) that atoms and bits occupy ontologically separate worlds; and, if we take for granted that humans originate in the physical world (because the physical matter predates the digital information in historical terms); then, it is natural to view the digital world as something strange and foreign. In fact, this is why the frontier metaphor has so often been invoked to describe “the virtual”: complete separation implies that, in order to enter “the virtual,” we need to exit the comfortable boundaries of the physical world. Often we imagine being dissolved (as in Tron), or sucked through the screen (as in Poltergeist, Stay Tuned, or Pleasantville). Of course, such scenarios are absurd.  Atoms and bits do not occupy ontologically separate worlds; rather, they are thoroughly and inextricably enmeshed.

The designers who coined the various single reality vocabulary terms, on the other hand, simply had to acknowledge the enmeshment of atoms and bits in order to describe what the digital technologies they created were actually doing. Motivated by the practical concerns of technological development, these folks made the earliest attempts to articulate causal linkages between the physical and the digital–aka, to show that the physical and the digital are in fact linked, and do indeed affect each other. These designers were therefore some of the first people to counter what we now call “digital dualism.”

Where Does the Single Reality Vocabulary Come From?

As early as 1991, Michael Naimark actively avoids reducing “reality” to atoms alone (even though he is writing on how the digital could come to simulate the physical). He does so by referring to the physical world not as “reality,” but as “unmediated reality.” There is a problematic assumption inherent in Naimark’s term, however, which is that atoms cannot constitute media. It is hard to imagine anyone accepting the idea that a piece of paper, a stone tablet, a vinyl record, or a braille sign is not a medium. Nonetheless, Naimark’s work represents an important conceptual expansion of what kinds of things constitute reality.

Milgram and Kishino (1994) attempt a more systematic deconstruction of the physical-digital binary. They argue that “the real” and “the virtual” exist as the two poles on a “mixed reality” continuum, and use the term “augmentation” to describe the intermediated spaces where atoms and bits mix. Milgram and Kishino (re)define “augmented reality” (a term already in use in 1994) to describe enmeshed phenomena in which the physical has priority, and coin the term “augmented virtuality” to describe enmeshed phenomena in which the digital has priority. Milgram and Kishino’s schema is laid out in the following diagram:

Milgram and Kishino’s theorization of mixed reality suffers from two related weaknesses. First, in their attempts to define “real” and “virtual,” Milgram and Kishino tautologically restate a commonly accepted binary distinction: “Real objects are any objects that have an actual objective existence… Virtual objects are objects that exist in essence or effect, but not formally or actually.” If that doesn’t seem to make sense, don’t worry: it’s not just you. Milgram and Kishino state that “real” objects are “actual,” and that “virtual” objects are not “actual,” but they make no attempts to tell us what “actual” actually is. Instead, they proceed to differentiate the “real” and the “virtual” not ontologically, but in design terms of “image quality” (read: resolution) and luminosity/opacity. Yet it is hard to imagine anyone accepting the idea that (say) jellyfish are not real, simply because they are transparent. As anyone who has swum into a swarm of them can tell you, jellyfish are indeed quite real (deceptive transparency notwithstanding).

The second major weakness of Milgram and Kishino’s approach is that they simply fail to embed their schema in existing philosophy pertaining to “the real” and “the virtual.” As Adriana de Souza e Silva and Daniel Sutko (2011) adeptly demonstrate, the meaning of the term “virtual” has shifted radically in recent decades; it has shifted more than that over recent centuries. Put simply, using the term “virtual” requires attention to nuance. Milgram and Kishino fail to take the term’s history into account, however, and rob it of any useful application by reducing its meaning to merely “not actual.”

Many designers have built upon Milgram and Kishino’s attempt to describe technologies that make physical and digital objects causally dependent on each other. “Mixed reality” concepts can be applied to a range of objects as simple as physical remotes that control digital displays, or digital remotes that control the physical movement of motors. A more sophisticated example of “mixed reality” technology is the range of “augmented reality” apps that combine inputs from cameras and sensors with digital information. Augmented reality navigation apps, for example, both take in information from physical space and guide movement within that space.

Milgram and Kishino’s “mixed reality” framework was certainly not the last effort by designers to describe the enmeshment of atoms and bit. Lifton and Paradiso (2008) have more recently described “dual reality,” in which “both real and virtual worlds are complete unto themselves, but also enhanced by the ability to mutually reflect, influence, and merge by means of sensor/actuator networks deeply embedded in everyday environments.” As do Milgram and Kishino, Lifton and Paradiso insist on ontological independence even though they acknowledge a causal linkage. (In other words, Lifton and Paradiso insist that the “real” and the “virtual” are distinct, separate worlds, but do acknowledge that those two worlds can affect each other. We’d call this “mild digital dualism”–but it’s still progress!)

Presumably, Lifton and Paradiso are motivated by the (legitimate) concern that, were they to say that “the real” and “the virtual” are co-produced, they would imply that physical matter cannot exist without digital information. Instead, Lifton and Paradiso settle on a model of “dual reality,” in which atoms and bits are separate things but are still able to influence each other.

As do Milgram and Kishino, Lifton & Paradiso generally fail to situate their taxonomy in the broader tradition of Western philosophy–and by so doing, avoid any engagement with the social and political consequences of their ontological claims. In fact, they appear to to be more concerned with technology and the world in-itself than with the humans who experience and inhabit that world. (This trend would begin to change, however, as human-centered design became more in vogue.)

Brian Johnson (2002) offers a model of blended reality that focuses primarily on human experience. He claims that designers have been hampered by the perception that the digital and the physical are separate, and argues that “this approach is fundamentally flawed and suggest[s] a more fruitful approach [could be achieved] through understanding the blending of physically and digitally mediated place.”

Johnson starts with the assumption that knowledge is derived from sensory experience. For this reason, he claims people have a tendency to spatialize knowledge. When we apply this spatializing tendency to digital information, we want to imagine our bodies somehow being in a space of digital information. He explains that virtual environments are not unlike dreams, memories, or fantasies, in that all of these allow us to experience sensations from stimuli that are not physically present. For Johnson, it is only experience that matters; where experience originates is irrelevant. As he puts it, “In the end, experience is experience. The qualification “virtual” is misleading and unnecessary.”

As the digital and the physical become further integrated, Johnson suggests that what matters is not the origin of any particular experience, but how the physical and the digital can be used in combination with each other to create environments that optimize human experience:

“[W]e already inhabit cyberspace, that it is not a case of choosing between reality and virtual reality: rather than striving for a substitute virtual experience, we should concentrate on strengthening and broadening the connections between the physical and digital aspects of our existence… [T]he central question becomes: what are appropriate, available, and desirable alterations of the built environment which will help occupants enjoy their activities more, accomplish their tasks more quickly or comfortably, and so on.”

Though Johnson is more philosophical in his approach to debunking digital-physical dualism (he adopts what we might call a phenomenological approach), he is still ultimately guided by the practical concerns of building environments–and therefore shies away from the social and political implications of the concepts he introduces.

While the single reality models offered by designers Milgram and Kishino, Lifton and Paradiso, and Johnson are probably sufficient for the practical work of building technology, these models are not the end of the story for single reality vocabulary terms. More recently, these terms have made what we describe as a “sociological turn.” The sociological turn is taking place because once the technologies single reality terms were coined to describe began to move out of design studios into our daily lives, certain social and political questions became unavoidable.

Above all, we find ourselves asking: If we now understand the world to be a hybrid of the physical and the digital, then how are we to understand the subjects who live in such a world? Are we now hybrids, too? Does our subjectivity now extend beyond the confines of our physical body? (Has our subjectivity perhaps always extended beyond our physical body?) Does a hybrid reality lead its hybrid subjects to adopt norms and practices that are fundamentally different from the norms and practices of previous generations? The designers above did not consider these questions, and yet they still give us the language through which we search for answers.

What we’ve called “the sociological turn” in single reality discourse marks the subject’s first entrance into debates about how to describe the conditions of our present-day reality. We will explore this turn in depth in Part Two.

Whitney Erin Boesel and PJ Rey have been sucked into the “world” through your screen! You can observe them there by opening Twitter: They’re @phenatypical and @pjrey.