NOTAS LEV MANOVICH. "THE LANGUAGE OF NEW MEDIA"

Linz, Austria, 1995. I am at Ars Electronica, the world’s most prestigious annual computer art festival. This year it drops the “Computer Graphics” category, replacing it with the new “net art” category, signaling a new stage in the evolution of modern culture and media. The computer, which since the early 60s was used as a production tool, has become a universal media machine: a tool used not only for production, but also for storage, distribution and playback. The World Wide Web crystallized this new condition; on the level of language, it was recognized already around 1990 when the term “digital media” came to be used along with “computer graphics.” At the same time, along with existing cultural forms, during the 1990s computers came to host an array of new forms: Web sites and computer games, hypermedia CD-ROMs and interactive installations—in short, “new media.” And if in 1985 I had to write a long computer program in a specialized computer language just to put a picture of a shaded cube on a computer screen, ten years later I can choose from a number of inexpensive, menu-based 3D software tools which run on ordinary PCs and which come with numerous ready-made 3D models, including detailed human figures and heads.

(31)

What had also come by 1995 was Internet—the most material and visible sign of globalization. And, by the end of the decade, it has also become clear that the gradual computerization of culture will eventually transform all of it. So, to invoke the old Marxist model of base and superstructure, if the economic base of modern society from the 1950s onward started to shift toward a service and information economy, becoming by the 1970s a so-called “post-industrial society” (Daniel Bell), and then later a “network society” (Manual Castells), by the 1990s

the superstructure started to feel the full impact of this change.4 If the “post- modernism” of the 1980s was the first, preliminary echo of this shift still to

come—still weak, still possible to ignore—the 1990s’ rapid transformation of culture into e-culture, of computers into universal culture carriers, of media into new media, demanded that we rethink our categories and models.

(32)

I use it (NEW MEDIA concept) as an umbrella term to refer to a number of various conventions used by designers of new media objects to organize data and structure user’s experience.

In this book I analyze the language of new media by placing it within the history of modern visual and media cultures. What are the ways in which new media relies on older cultural forms and languages and what are the ways in which it breaks with them? What is unique about how new media objects create the illusion of reality, address the viewer, and represent space and time? How do conventions and techniques of old media—such as the rectangular frame, mobile viewpoint and montage—operate in new media? If we are to construct an archeology which will connect new computer-based techniques of media creation with previous techniques of representation and simulation, where should we locate the essential historical breaks?

(34)

Myst and Doom, Jurassic Park and Titanic, to the works of international new media artists and collectives such as ART+COM, antirom, jodi.org, George Legrady, Olga Lialina, Jeffrey Shaw, and Tamas Waliczky.

The computarization of culture

TOPICS OF THE BOOK

·       the parallels between cinema history and the history of new media;

·       the identity of digital cinema;

·       the relations between the language of multimedia and nineteenth century pre-

cinematic cultural forms;

·       the functions of screen, mobile camera and montage in new media as

compared to cinema;

·       the historical ties between new media and avant-garde film.

"digital materialism." Rather than imposing some a priori theory from above, I build a theory of new media from the ground up. I scrutinize the principles of computer hardware and software, and the operations involved in creating cultural objects on a computer, in order to uncover a new cultural logic at work.

(35)

The section “Principles of New Media” describes four key trends which, in my view, are shaping the development of new media over time: modularity, automation, variability and transcoding.

Just as avant-garde filmmakers throughout cinema's existence offered alternatives to its particular narrative audio-visual regime, the task of avant-garde new media artists today is to offer alternatives to the existing language of computer media. This can be better accomplished if we have a theory of how "mainstream" language is structured now and how it is evolving over time.

(36)

LANGUAGE, OBJECT AND REPRESENTATION

I felt justified in using the word “language” to signal the different focus of this work: the emergent conventions, the recurrent design patterns, and the key forms of new media. I considered using the words “aesthetics” and “poetics” instead of “language,” eventually deciding against them. Aesthetics implies a set of oppositions which I would like to avoid— between art and mass culture, between the beautiful and the ugly, between the valuable and the unimportant. Poetics also brings with it undesirable connotations.

In addition, I hope to activate connotations which accompanied the use of the word “object” by the Russian avant-garde artists of the 1920s. Russian Constructivists and Productivists referred to their creations as objects (“vesh,” “construktsia,” “predmet”) rather than works of art. Like their Bauhaus counterparts, they wanted to take on the roles of industrial designers, graphic designers, architects, clothing designers and so on, rather than remain fine artists producing one-of-a kind works for museums or private collections. The word pointed toward the model of industrial mass production rather than the traditional artist’s studio, and it implied the ideals of rational organization of labor and engineering efficiency which artists wanted to bring into their own work.

(39)

Today, those few who are able to resist the temptation to immediately create an “interactive CD-ROM,” or to make a feature length “digital film,” and instead are able to focus on determining the new media equivalent of a shot, a sentence, a word, or even a letter, are rewarded with amazing findings.

(40)

In this book I suggest that not only individual new media objects, but also the interfaces, both of an operating system and of commonly used software applications, also act as representations. That is, by organizing data in particular ways and by making it possible to access it in particular ways, they privilege particular models of the world and of the human subject. For instance, the two key

ways to organize computer data commonly used today — a hierarchical file system (Graphical User Interface from 1984 Macintosh onward) and a “flat,” non- hierarchical network of hyperlinks (1990s World Wide Web) — represent the world in two fundamentally different and, in fact, opposing ways. Hierarchical file system assumes that the world can be reduced to a logical and hierarchical order, where every object has a distinct and well defined place.

I define screen as a rectangular surface which frames a virtual world and which exists within the physical world of a viewer without completely blocking her visual field.

Simulation refers to technologies which aim to completely “immerse” the viewer within the virtual universe: Baroque Jesuit churches, nineteenth century panorama, twentieth century movie theaters.

Representation — control (“Cultural Interfaces” section). Here I oppose an image as a representation of an illusionary fictional universe and an image as a simulation of a control panel

Representation — action representational technologies used to enable action, i.e. to allow the viewer to manipulate reality through representations (maps, architectural drawings, x-ray, telepresence). I refer to images produced by later technologies as image-instruments.

Representational technologies allow for creation of traditional aesthetic objects, i.e. something which is fixed in space or time and which refers to some referent(s) outside itself. By foregrounding the importance of person-to- person telecommunication, and “tele”-cultural forms in general which do not produce any objects, new media forces us to reconsider the traditional equation between culture and objects.

(41)

Simulation refers to various computer methods for modeling other aspects of reality beyond its visual appearance: movement of physical objects, shape changes over time in natural phenomena (water surface, smoke), motivations, behavior, speech and language comprehension in human beings.

WHAT’S NEW MEDIA?

Distribution

Production

Exhibition

Shall we accept this definition? If we want to understand the effects of computerization on culture as a whole, I think it is too limiting. There is no reason to privilege computer in the role of media exhibition and distribution machine over a computer used as a tool for media production or as a media storage device. All have the same potential to change existing cultural languages. And all have the same potential to leave culture as it is.

(43)

new media represents a convergence of two separate historical trajectories: computing and media technologies. Both begin in the 1830's with Babbage's Analytical Engine and Daguerre's daguerreotype.

The synthesis of these two histories? The translation of all existing media into numerical data accessible for computers. The result is new media: graphics, moving images, sounds, shapes, spaces and text which become computable, i.e. simply another set of computer data.

ather than focusing on familiar categories such as interactivity or hypermedia, I suggest a different list. This list reduces all principles of new media to five: numerical representation, modularity, automation, variability and cultural transcoding.

(44)

Around 1800, J.M. Jacquard invented a loom which was automatically controlled by punched paper cards. The loom was used to weave intricate figurative images, including Jacquard's portrait.

(45)

We should not be surprised that both trajectories — the development of modern media, and the development of computers — begin around the same time. Both media machines and computing machines were absolutely necessary for the functioning of modern mass societies. The ability to disseminate the same texts, images and sounds to millions of citizens thus assuring that they will have the same ideological beliefs was as essential as the ability to keep track of their birth records, employment records, medical records, and police records. Photography, film, the offset printing press, radio and television made the former possible while computers made possible the latter. Mass media and data processing are the complimentary technologies of a modern mass society; they appear together and develop side by side, making this society possible.

(46)

Universal Turing Machine

the development of a suitable storage medium and a method for coding data represent important parts of both cinema and computer pre-histories.

in the same year, 1936, the two trajectories came even closer together. Starting this year, and continuing into the Second World War, German engineer Konrad Zuse had been building a computer in the living room of his parents' apartment in Berlin. Zuse's computer was the first working digital computer. One of his innovations was program control by punched tape. The tape Zuse used was

actually discarded 35 mm movie film.15

(47)

The pretense of modern media to create simulation of sensible reality is similarly canceled; media is reduced to its original condition as information carrier, nothing else, nothing more. In a technological remake of the Oedipal complex, a son murders his father. The iconic code of cinema is discarded in favor of the more efficient binary one. Cinema becomes a slave to the computer.

(48)

The two separate historical trajectories finally meet. Media and computer — Daguerre's daguerreotype and Babbage's Analytical Engine, the Lumière Cinématographie and Hollerith's tabulator — merge into one. All existing media are translated into numerical data accessible for the computers. The result: graphics, moving images, sounds, shapes, spaces and text become computable, i.e. simply another set of computer data. In short, media becomes new media.

No longer just a calculator, a control mechanism or a communication device, a computer becomes a media processor.

No longer just an Analytical Engine, suitable only to crunch numbers, the computer became Jacqurd's loom — a media synthesizer and manipulator.

PRINCIPLES OF NEW MEDIA

1. Numerical Representation

All new media objects, whether they are created from scratch on computers or converted from analog media sources, are composed of digital code; they are numerical representations.

media becomes programmable.

As I will show below, new media follows, or actually, runs ahead of a quite a different logic of post-industrial society — that of individual customization, rather that of mass standardization.

(52)

2. Modularity

This principle can be called "fractal structure of new media.” Just as a fractal has the same structure on different scales, a new media object has the same modular structure throughout. Media elements, be it images, sounds, shapes, or behaviors, are represented as collections of discrete samples (pixels, polygons, voxels, characters, scripts). These elements are assembled into larger-scale objects but they continue to maintain their separate identity. The objects themselves can be combined into even larger objects -- again, without losing their independence. For example, a multimedia "movie" authored in popular Macromedia Director software may consist from hundreds of still images, QuickTime movies, and sounds which are all stored separately and are loaded at run time. Because all elements are stored independently, they can be modified at any time without having to change Director movie itself. These movies can be assembled into a larger "movie," and so on.

3. Automation

Numerical coding of media (principle 1) and modular structure of a media object (principle 2) allow to automate many operations involved in media creation, manipulation and access. Thus human intentionally can be removed from the

creative process, at least in part.19

(52 )

computer games are highly codified and rule- based

By the end of the twentieth century, the problem became no longer how to

create a new media object such as an image; the new problem was how to find the object which already exists somewhere. That is, if you want a particular image, chances are it is already exists -- but it may be easier to create one from scratch when to find the existing one. Beginning in the nineteenth century, modern society developed technologies which automated media creation: a photo camera, a film camera, a tape recorder, a video recorder, etc. These technologies allowed us, over the course of one hundred and fifty years, to accumulate an unprecedented amount of media materials: photo archives, film libraries, audio archives...This led to the next stage in media evolution: the need for new technologies to store, organize and efficiently access these media materials. These new technologies are all computer-based: media databases; hypermedia and other ways of organizing media material such the hierarchical file system itself; text management software; programs for content-based search and retrieval. Thus automation of media access is the next logical stage of the process which was already put into motion when a first photograph was taken. The emergence of new media coincides with this second stage of a media society, now concerned as

much with accessing and re-using existing media as with creating new one.2

(55)

4. Variability

55

A new media object is not something fixed once and for all but can exist in different, potentially infinite, versions. This is another consequence of numerical coding of media (principle 1) and modular structure of a media object (principle 2). Other terms which are often used in relation to new media and which would be appropriate instead of “variable” is “mutable” and “liquid.”

New media, in contrast, is characterized by variability. Instead of identical copies a new media object typically gives rise to many different versions. And rather being created completely by a human author, these versions are often in part automatically assembled by a computer.

(56)

in a computer age database comes to function as a cultural form of its own. It offers a particular model of the world and of the human experience. It also affects how the user conceives of data which it contains.

the principle of variability is a useful in allowing us to connect many important characteristics of new media which on first sight may appear unrelated. In particular, such popular new media structures as branching (or menu) interactivity and hypermedia can be seen as particular instances of variability principle (4.4 and 4.5, respectively). In the case of branching interactivity, the user plays an active role in determining the order in which the already generated elements are accessed. This is the simplest kind of interactivity; more complex kinds are also possible where both the elements and the structure of the whole object are either modified or generated on the fly in response to user's interaction with a program. We can refer to such implementations as open interactivity to distinguish them from the closed interactivity which uses fixed elements arranged in a fixed branching structure. Open interactivity can be implemented using a variety of approaches, including procedural and object- oriented computer programming, AI, AL, and neural networks.

Hypermedia, the other popular structure of new media, can also be seen as a particular case of the more general principle of variability. According to the definition by Halacz and Swartz, hypermedia systems “provide their users with the ability to create, manipulate and/or examine a network of information-

containing nodes interconnected by relational links.”29 Since in new media the individual media elements (images, pages of text, etc.) always retain their

individual identity (the principle of modularity), they can be "wired" together into more than one object. Hyperlinking is a particular way to achieve this wiring

(59)

In programming, there is clear separation between algorithms and data. An algorithm specifies the sequence of steps to be performed on any data, just as a hypermedia structure specifies a set of navigation paths (i.e., connections between the nodes) which potentially can be applied to any set of media objects.

(60)

More generally, every hypertext reader gets her own version of the

complete text by selecting a particular path through it. Similarly, every user of an interactive installation gets her own version of the work. And so on. In this way new media technology acts as the most perfect realization of the utopia of an ideal society composed from unique individuals. New media objects assure users that their choices — and therefore, their underlying thoughts and desires — are unique, rather than pre-programmed and shared with others. As though trying to compensate for their earlier role in making us all the same, today descendants of the Jacqurd's loom, the Hollerith tabulator and Zuse's cinema-computer are now working to convince us that we are all unique.

(61)

While I deduced the principle of variability from more basic principles of new media — numerical representation (1) and modularity of information (2) — it can also be seen as a consequence of computer’s way of to represent data and model the world itself: as variables rather than constants. As new media theorist

61

and architect Marcos Novak notes, a computer — and computer culture in its

wake — substitute every constant by a variable.33

(62)

Cinema is not the only media technology which, emerging towards the end of the nineteenth century, employed a discrete representation. If cinema sampled time, fax transmission of images, starting in 1907, sampled a 2D space; even earlier, first television experiments (Carey, 1875; Nipkow, 1884) already involved

sampling of both time and space.36 However, reaching mass popularity much earlier than these other technologies, cinema is the first to make the principle of a discrete representation of the visual a public knowledge.

Before computer multimedia became commonplace around 1990, filmmakers were already combining moving images, sound and text (be it intertitles of the silent era or the title sequences of the later period) for a whole century. Cinema thus was the original modern "multimedia." We can also much earlier examples of multiple-media displays, such as Medieval illuminated manuscripts which combined text, graphics and representational images.

For example, once a film is digitized and loaded in the computer memory, any frame can be accessed with equal ease. Therefore, if cinema sampled time but still preserved its linear ordering (subsequent moments of time become subsequent frames), new media abandons this "human-centered" representation altogether — in order to put represented time fully under human control. Time is mapped onto two-dimensional space, where it can be managed, analyzed and manipulated more easily.

(67)

sed in relation to computer-based media, the concept of interactivity is a tautology. Modern human-computer interface (HCI) is by its very definition interactive. In contrast to earlier interfaces such as batch processing, modern HCI allows the user to control the computer in real-time by manipulating information displayed on the screen. Once an object is represented in a computer, it automatically becomes interactive. Therefore, to call computer media interactive is meaningless -- it simply means stating the most basic fact about computers.

Rather than evoking this concept by itself, in this book I use a number of other concepts, such as menu-based interactivity, salability, simulation, image- interface, and image-instrument, to describe different kinds of interactive structures and operations. The already used distinction between “closed” and “open” interactivity is just one example of this approach.

(71)

Also like Blade Runner, GUI vision also came to influence many other areas of culture. This influence ranges from purely graphical (for instance, use of GUI elements by print and TV designers) to more conceptual. In the 1990s, as the Internet progressively grew in popularity, the role of a digital computer shifted

from being a particular technology (a calculator, a symbol processor, an image manipulator, etc.) to being a filter to all culture, a form through which all kinds of cultural and artistic production is being mediated. As a window of a Web browser comes to replace cinema and television screen, a wall in art gallery, a library and a book, all at once, the new situation manifest itself: all culture, past and present, is being filtered through a computer, with its particular human-computer

interface.

(76)

The second section “The Screen and the User” discusses the key element of the modern interface — the computer screen. As in the first section, I am interested in analyzing continuities between a computer interface and older cultural forms, languages and conventions. The section positions the computer screen within a longer historical tradition and it traces different stages in the development of this tradition: the static illusionistic image of Renaissance painting; the moving image of film screen, the real-time image of radar and television; and real-time interactive image of a computer screen.

(79)

The term human-computer interface (HCI) describes the ways in which the user interacts with a computer. HCI includes physical input and output devices such a monitor, a keyboard, and a mouse. It also consists of metaphors used to conceptualize the organization of computer data. For instance, the Macintosh interface introduced by Apple in 1984 uses the metaphor of files and folders arranged on a desktop. Finally, HCI also includes ways of manipulating this data, i.e. a grammar of meaningful actions which the user can perform on it. The example of actions provided by modern HCI are copy, rename and delete file; list the contents of a directory; start and stop a computer program; set computer’s date and time.

The term HCI was coined when computer was mostly used as a tool for work. However, during the 1990s, the identity of computer has changed. In the beginning of the decade, a computer was still largely thought of as a simulation of a typewriter, a paintbrush or a drafting ruler -- in other words, as a tool used to produce cultural content which, once created, will be stored and distributed in its appropriate media: printed page, film, photographic print, electronic recording. By the end of the decade, as Internet use became commonplace, the computer's public image was no longer that of tool but also that a universal media machine, used not only to author, but also to store, distribute and access all media.

universal media machine

(80)

Cinema, the printed word and human-computer interface: each of these traditions has developed its own unique ways of how information is organized, how it is presented to the user, how space and time are correlated with each other, how human experience is being structured in the process of accessing information. Pages of text and a table of contents; 3D spaces framed by a rectangular frame which can be navigated using a mobile point of view; hierarchical menus, variables, parameters, copy/paste and search/replace operations -- these and other elements of these three traditions are shaping cultural interfaces today. Cinema, the printed word and HCI: they are the three main reservoirs of metaphors and strategies for organizing information which feed cultural interfaces.

(82)