Big, Bigger, Biggest:
Inventing Systems for Immense Digital Images (and Beyond)
David Rogers, Carl Diegert, Clifford Ross, Michael Hawley, Bran Ferren, Eric Rosenthal
Mountain IV, by Clifford Ross, an image consisting of a billion pixels.
When digital pictures get really, really big – more than a billion pixels – they demand new system architectures and invite new modes of interaction. How can you capture a multi-gigapixel snapshot? How do you display it on a really large wall? How do you interact with it? What are the underlying system architectures for managing such large images?
This memo offers two things. First, a synopsis of the “Big Picture Summit,” convened in December 2004, chalks out the technical, psychophysical and artistic challenges of enormous digital images. And second, a straw-man proposal offers a longer-range plan for realizing a significant new kind of “Data Observatory,” and with it, the intermediate thrusts and partnerships to get there.
The image “Mountain IV” (frontispiece) was created by Clifford Ross. It is one of the biggest digital landscape images in the world. Most people encounter it as a mural - a 5 ft. x 10 ft. single print. The resolution and detail is astonishing, and the experience of seeing it, for most viewers, is quite staggering.
Ross is pushing the limits of both analog and digital technology to create a ‘you are there’ experience with his high-resolution landscape photographs.  He ran into a number of technical hurdles that exposed underlying system needs for dealing with really large, data heavy images. Most obvious were a need to capture images like this digitally and a need to display them in such a way as to reveal all their detail in a way which would enable people to interact with them in a fluid way. This display system would clearly not be a conventional print, a laptop or desktop computer display: a new kind of presentation is needed – one that promotes encounters with truly vast digital imagery.
Ross’ artistic needs intersected with those of the scientific community. As a first step, a unique group of talented artists, engineers and scientists have assembled to meet the challenge.
Ross wants to “fill the eye with so much information that it overflows and reaches the heart.” Similarly, scientists and engineers want to understand the science behind extremely large, data-intensive images. Intuition and understanding are the cornerstones of scientific achievement, and scientists are reaching out to other disciplines for help in understanding data. Facilities such as the VIEWS Corridor at Sandia National Laboratories display high-resolution images of interest to scientific, engineering and national security personnel. Viewing high-resolution images in a setting that promotes interaction between people is a critical need for industry and government. It also coincides with Ross’ artistic drive to present images with what he calls the highest possible “reality quotient”.
“Art is emotional, but the path is technical,” says Ross. From capturing a gigapixel image of a moment in time, to an ultimate museum-quality experience of the image, there are many unsolved technical challenges. We want to remove technical obstacles to producing compelling images such as “Mountain IV,” so that Big Picture display becomes more easily achievable. Recognizing that technology and new consumer products will continue to impact Big Picture issues, it is essential to create and support a dynamic, highly-skilled team of artists and technologists dedicated to Big Picture vision. As a first step, this Big Picture team will create a never-before-seen viewing installation for digital images that is so compelling and natural that people will be drawn into the images without being impeded by the technology displaying them.
2. The Experience
For artists, creating large scale, detailed images is meaningful only if they can create a proper viewing experience that doesn’t distract the viewer from the impact of the image itself. For scientists, large scale, detailed images must be presented in a manner that will enable interaction, trigger intuition, and lead to scientific discovery. Of significant concern is how to display and interact with these images, overcoming the obstacles created by their sheer size.
|Figure 1: Roy Lichtenstein, Mural with Blue Brushstrokes, 1989 68 x 32 ft. New York City.|
It is important to understand how one physically encounters large images. Consider a large painting such as Lichtenstein’s Mural with Blue Brushstrokes of 1986 (Fig. 1), which hangs in the lobby of the Equitable Tower at 787 Seventh Ave. in New York City. In Figure 1, note the size of a person standing on a ladder. This painting is a large, public image, meant to exist at the scale of the lobby. It is not to be approached or inspected.
In contrast to this experience, scientists at Sandia National Laboratories’ VIEWS Corridor can walk right up to a projected digital image and examine details within it (Fig. 2). To get overall context around that detail, they simply step back and take it all in. This ‘zoom and pan by walking’ is a very effective way for groups of people to navigate large images. Because Sandia’s venue has no fixed seating, the venue itself encourages people to walk around and investigate the image independently. This is an important feature of a facility like the VIEWS Corridor, because participants communicate to each other both verbally and non-verbally when they experience the image. Colleagues discussing a specific detail gather in front of it, and others in the room know where the area of interest is, even if they are not part of the group investigating it.
Big Picture images, though large, are meant to be intimately encountered, inspected and discussed. Part of the experience of viewing a Big Picture is the high resolution, and the concomitant experience of being drawn in – past the impact of the entire image to its smallest details. The images invite intimate inspection, as details are revealed as one moves closer to the image.
The experience is everything. Large digital images, whether they are for scientific or artistic ends, must allow for proper and satisfying interaction.
Figure 2: The VIEWS Corridor at Sandia National Laboratories, where scientists study high-resolution renderings of simulation data.
3. The Big Picture Overview
Clifford Ross uses his patented R1 camera to capture landscape images on Kodak aerial film. This one-of-a-kind camera uses large format (9 in. x 18in.) negatives, which are then scanned at high resolution. Ross and his technical team then manipulate the resulting digital file with Adobe Photoshop running on Apple computers. The process of building adjustment layers can boost the size of the working file to 22 gigabytes. Over the course of six months, the image is painstakingly massaged into final form. Twelve to fourteen full-scale proofs are printed on Kodak Endura chromogenic paper with an Oce Lightjet laser printer. The final result is displayed in a unique framing system of Ross’ design, which allows intimate and satisfying perusal of the finest details in the image. The frame systems include the largest sheets of non-reflective glass ever made, manufactured by Tru-Vue to his specifications, minimizing distracting reflections across the entire image.
Ross’ stated goal is to produce an image firmly grounded in the realistic detail which his camera provides, but consistent with his memory of the scene as he experienced it. This, of course, is a subjective consideration. It differs from a scientific approach by virtue of his willing embrace of subjective interpretation to further his artistic goal. But Summiteers have stressed that the difference between an artist’s interpretive role compared to that of a scientist is one only of intent. Every image of the universe released by NASA and the Hubble Space Telescope Science Institute is an interpreted image. The scientists aim to render, through subjective interpretive means, the singular identity of the multitude of objects depicted in outer space, as well as the sense of wonder that they wish to share with the viewer.
This synergy between artistic and scientific goals of images is a central reason for Big Picture collaboration. Zoltan Levay, Imaging Resource Lead of the Hubble Space Telescope Science Institute at Johns Hopkins University, commented on this topic, stating that the “fundamental use of our [public Hubble] images is to disseminate significant science findings from Hubble, so we certainly don't want to manipulate the images in such a way that would mislead. Yet, we know that an unattractive image that repels viewers is counter-productive. So we struggle to balance these sometimes-contradictory goals.”
3.1 Image Capture
There are many digital image capture technologies available, from consumer cameras for taking family photos to custom-built CCD arrays for capturing astronomical data. Today, one can buy a high-end consumer digital camera such as the Nikon D2X for about $5,000. This camera captures over 12 million pixels in a single image.
Summit participants felt that in 10 years, there will likely be digital consumer cameras that meet the needs of Big Picture capture. It is the intention of the BPS group to return to the issue of image capture after pushing display technology to fulfill the needs of existing film capture technology. The long-term solution to Ross’s goal of “instantaneous gigapixel capture” (IGC) is clearly digital. In the short term, the best method to capture gigapixel images is the use of large format film cameras, such as the R1.
Though digital photography is a long-term solution, the summit identified several areas for Ross to explore in the short term, all of which could enrich his current images:
- Increase the dynamic range of the capture.
- Increase the bit depth of scans.
- Increase color space beyond red, green and blue. Full visible spectrum capture and display.
- Capture information outside the normal range of sight. For example, infrared.
- Decrease the size of the negative to 8”x10” to take advantage of state-of-the-art cameras, lenses, films, and scanning technologies not available for the larger format of the R1.
- Increase the resolution of the scans per square inch.
Summiteers noted that image capture should be re-evaluated periodically, to take advantage of appropriate emerging technologies.
Manipulation is perhaps the best-served portion of the Big Picture , because Hollywood and the digital photography industry already have such high demands for tools.
The Big Picture group will aggressively exploit leading edge commodity technology such as Photoshop. Faster computers are always welcome, and large, high-resolution displays would improve the process, and take better advantage of the artists' time.
Again, Summiteers noted that image manipulation should be re-evaluated periodically, to take advantage of appropriate emerging technologies.
3.3 The Viewing Experience
Arguably more technically challenging, and certainly more psycho-physically challenging than image capture, is the challenge of creating a successful way to view these images at full resolution.
One challenge is how to digitally display a gigapixel image at full resolution. In other words, how can we show Ross’s “Mountain” images so that we can see every pixel created by the scan? Sandia’s VIEWS Corridor has 48 projectors, each of which displays approximately 1.3 million pixels. The resulting display wall is approximately 42 ft. wide by 10.2 high. The total resolution of the display wall is 62 megapixels, approximately 1/16th the resolution needed to display a gigapixel . From the Sandia facility, we see that the dimensions of a full-resolution gigapixel display wall would be daunting. Big Picture participants are working on ‘the resolution problem’ – determining how tightly we can pack pixels onto a wall. This is largely a technical issue, taking into account the size, accessibility, and cooling needs of the projectors.
Projectors considered for the Big Picture display have defining characteristics, such as brightness, contrast, color range, resolution, portability, and cost. Currently, Sandia National Laboratories, and a number of the Big Picture participants are evaluating a variety of high-end projection technologies, including a yet-to-be-released laser projector from Sony. This evaluation will continue throughout the Big Picture project, as new technologies become available. If we determine that a full-resolution gigapixel display is undesirable based on mitigating factors, we can reverse engineer the facility to create the maximum size display that is comfortable and satisfying to experience and see how many pixels will fit.
Part of the encounter with the image may then become one of making the full depth of information in the image available to the viewer by secondary means. This leads naturally to future work. If we can make more pixels available, can we also offer other types of information? Can we offer an interactive setting, in which others can add content as well? We address these issues in Section 4.4.
There are several current proposals for how we physically encounter large digital images.
3.3.1 Wall Projection
The ‘image-on-the-wall’ approach is a rear-projection display built into a wall. The viewer encounters it in a traditional way, like a mural. This is the solution used by Sandia National Laboratories at the VIEWS facility. People viewing the image can literally put their noses against the parts of it that they are able to reach, but the lower and upper parts of the image are difficult or impossible to reach. (In fact, the nose-to-wall experience at the VIEWS facility is not wholly satisfying even with the parts of the wall that are accessible because at that close proximity the image dissolves into pixels.) The ability of the viewer to interact with other viewers in front of the image, take in the image as a whole unit, and investigate the parts of the image that are within reach, is very good. The experience conforms to normal human interaction.
3.3.2 Ceiling Projection
The use of the ceiling as the display surface for rear projection, Ross’s ‘Digital Sistine Ceiling’ concept, (see section 4.2), which may be very effective for artistic purposes, may be less so in regard to general display. If viewers walk into a room with an illuminated ceiling, no doubt their surprise will prime them for a unique event. The image being in an unusual location signifies to the viewer that they are going to have a special experience. Across cultures, looking up signifies hope, the heavens, spirituality, and harkens back to ceiling paintings in churches and palaces by artists like Michelangelo and Tiepolo. Although ceiling display may prime the viewer in an appropriate way for an artistic experience, it will no doubt disorient them if the purpose is more related to data comprehension.
From a direct, experiential point of view, ceiling display will allow viewers to move across the entire image, with no one portion of the image further out of reach than another. The “top” of a large image will not be seen from a further viewing distance than the “center” or “bottom” of the image, as would be the case in looking at a 15’ x 30’ wall display. If the ceiling display was 9.5’ from the floor, viewers could significantly move closer to the image with the aid of a simple stool or stepladder. And one can imagine lying down on the floor to gain greater viewing distance, an experience not so uncommon in certain art museum installations.
3.3.3 Floor Projection
In conjunction with Northrop Grumman, Applied Minds, Inc. has designed and built a 20 megapixel floor display with projectors suspended overhead in a grid. It is an immediate “map-on-the-floor” experience, and it is easy to inspect any part of the image closely, simply by getting down on one’s hands and knees. With a limited number of people “on” the image, shadows are not a significant problem because of the multitude of overhead projectors. Stepping back from the image is difficult, but can be achieved through a series of platforms surrounding the projection area, creating an amphitheatre-like space.
4. BPS Goals/Next Steps
Inventing a new Big Picture system will be an iterative process. The multitude of perceptual, artistic and technological challenges mean that this must be a ‘hands-on’ development process in which the group works directly with emerging technologies and designs. Big Picture Summiteers propose a working group which serves as the collaborative center for artists, engineers, scientists, and businesspeople to explore issues of interest all those thinking about the Big Picture.
This collaborative approach leverages the Summiteers’ combined experience – from building high-resolution displays  to creating evocative digital records of entire countries .
BPS will partner with leading individuals, institutions, and corporations in the high-technology sector, the arts, and the scientific community. We propose a long-range vision and framework for a ‘Data Observatory’, as well as a series of short-range steps to get there.
4.1 Studio Space
In the short term, BPS will establish a shared, low-cost space that allows collaborators to set up and explore technology, designs, and artistic content. Participation by a wide range of artists and scientists is essential to this effort. The BPS Studio Space is envisioned as a studio containing projectors, screens, computers, and other hardware and software necessary for the display of images. Sandia National Laboratories is working with interested vendors, such as Sony so that emerging projection and display technologies would be on hand for experimentation within the studio.
4.2 A “Digital Sistine Ceiling”
The first Big Picture Summit project (BPS 1) is the ‘Digital Sistine Ceiling’. This is envisioned as a mobile, room-sized installation with a high-resolution display on the ceiling (Fig. 3). This traveling installation would be used for viewing a series of Ross’ “Mountain” images in art museums, but will also be useful as a display system for scientific images contributed by the BPS group. Although physical and technical considerations may dictate a smaller display, or even a shift from ceiling to wall or floor projection, it is our goal to design and build a system to display an entire gigapixel image.
Figure 3: BPS 1 “Digital Sistine Ceiling” Sketch
- The image must be seamlessly displayed. This means that the scope of each projector will not be detectable, and the material on which the image is projected will not interfere with the image in any way.
- The projection surface will be uninterrupted by structure, connectors or joints of any kind that the viewer can perceive.
- The entire installation must be mobile - it can be entirely disassembled and moved from one installation site to another.
- The installation must be sturdy enough to withstand many moves over several years.
- The image is rear-projected from above.
- The viewer should never see individual pixels. Necessary image resolution will be determined by what people can perceive at their closest access to the ceiling.
- There will be some way for people to get closer to the ceiling (step ladders, rolling stools, etc.)
Main Technical Challenges
- What is the surface material that the image is projected on, and how is that material suspended? What are the limitations on that material - for example, if it is glass, there are limits on the size of panel that can be shipped, or that can support its own weight.
- Which projectors will fill the various requirements for the large array of units that are to project the gigapixel images?
- What software is already available to control the multitude of computers and projectors in a tightly controlled manner?
- Determine whether wall or ceiling projection provides the best effect for this first, artistically oriented display system, and the minimum distance from which the viewer will see the image.
- Determine the pixels per foot formula for pixel free viewing, which will be dependent on the results of task 1.
- Determine what the maximum physical size of the image can be based on the results of tasks 1 and 2, taking into account architectural issues and physical limitations (i.e. how close can the projectors be placed to one another, and how will that impact the pixels per square foot formula?).
- Determine the optimum design for the software to be used in controlling the display.
- Determine the optimum computer hardware to be used in the display.
- Determine possible support structures.
- There shall be easy access to projectors for service and tuning.
- Sufficient budget should be allocated for repair/replacement of projectors, (including spares), replacement bulbs, and regular alignment and color matching of all projectors.
4.3 Toward a New Digital Image Architecture
A more ambitious undertaking of the BPS group will be a Big Picture Display with Interactive Digital Binoculars (BPS 2). Envisioned as a collaborative museum-quality display environment for extreme high-resolution images, it is an installation that invites deeper involvement from visitors.
Stepping into the BPS 2 exhibit space, the viewer is greeted by a bright image on the far wall, which displays a digital image. In front of the display, there is a table, holding a collection objects that look like binoculars. (Fig. 4) Some visitors to the space are already near the display, and several have the binoculars held to their eyes. They talk and gesture to each other as they look at the image.
Figure 4: BPS 2 New Digital Image Architecture with Interactive Digital Binoculars
The viewer approaches the display, and like a traditional piece of artwork, much of its detail is revealed. The experience of the viewer changes, however, when he picks up one of the digital binoculars on the table, and looks through it. This device is a digital magnifier, letting the viewer explore the image in a very different way. Point it at the image, and more detail is revealed.
This concept draws on the example of the simple tube offered to those viewing “The Heart of the Andes” by Frederick Church in the 19th century. The tube, without optics or lenses of any kind, allowed viewers to isolate details of his huge panoramic landscape, enriching and personalizing their experience by giving each visitor a device for inspecting the artwork himself.
In the BPS version of Frederick Church’s viewing tube, each toy consists of a small, light, high-resolution display and wireless communication hardware. Each toy communicates information about its position and orientation, and receives image data to show on its display. They are intended as intuitive tools that show all of the data in the Big Picture. Binoculars are familiar to most people, and using them is very simple – place them up against your eyes, and look through them to magnify the scene. Digital binoculars operate on the same principle, allowing personal interaction with details of the image – even the part of the image that can’t be shown on the wall without the aid of the binoculars.
Digital Binoculars serve several purposes:
- They allow people access to a richer array of data than what is available to them by simply standing in front of the Big Picture Display. This additional data can be higher resolution, and text or even video that has been 'placed' in the image.
- The binoculars should be hand held. In order to be inviting, they should be easy to use, ergonomic, and even beautiful.
- The devices should be un-tethered.
There are several other proposals for achieving this deeper penetration into the Big Picture Display, including interactive, intelligent ‘magnifiers’ on the wall that detect where a person is looking and magnify that portion of the image.
4.4 The “Data Observatory”
The long-range vision of the BPS goes beyond solving the technical and artistic challenges posed by Ross’ gigapixel images, the “Digital Sistine Ceiling” (BPS 1) and even the New Digital Image Architecture (BPS 2). The long-range vision calls for a “Data Observatory” – a facility for experiencing images, sound, and information in an additive environment that invites global participation (BPS 3). Building on the New Digital Image Architecture, we envision an internet-accessible facility that invites collaboration.
Images will serve as organizing structures for all types of annotation and content. A high-resolution image of a birdcage at a zoo might invite such additions as bird songs and ornithological data. Using browser-based interfaces, children could add their reactions and thoughts, while experts in a variety of fields could add all types of technical information, turning the image into an open source encyclopedia and communication center. Both large and small scale additions would be welcomed in this environment, with non-scientists and even children participating with their own contributions and reactions.
The digital binoculars and Internet interfaces associated with an image could filter specific information. For example, a high-resolution image of a forest could have filters designed to relay specific types of information of particular interest to the viewer. The “Data Observatory” would be a literal and virtual space that encourages discovery, promotes the exercise of intuition, and invites thoughtful interaction. The digital binoculars from BPS 2 could be redesigned to deliver sound to the viewer and deliver speech based information back “in” to the image.
Anyone, anywhere in the world could design a device or Internet interface that could be used to access the image. The necessary technology and communication protocols will be available and published on the web. People could create high-resolution 'Open Source' images that are posted on the Web, submit multi-media content such as sound, video, or text for posting within an existing image, and create high-resolution content for adding to the display of a digital canvas.
The “Data Observatory” would be a worldwide communications center - a global classroom where everyone can participate as both student and teacher.
5. BPS: A Not-for-profit Research and Educational Entity
The BPS group plans to evolve into a not-for-profit entity that may generate intellectual property (IP). Revenues resulting from that IP will be used in a manner similar to the formula worked out by Scripps Research Institute and other research centers. It will largely be used to support BPS efforts, but royalties will also be shared with participating scientists in order to attract the best talent by rewarding innovation. A significant component of our goal is to affiliate with an existing university to create an ongoing educational component to our work.
6. Funding and Exhibition
BPS will identify government agencies, corporations, foundations, universities, art museums, and individuals with a natural or inherent interest in seeing BPS1 succeed. In addition, BPS will seek to collaborate with those entities interested in hosting, financing, or promoting subsequent BPS work.
- Government agencies and corporations should find access to this group of top imaging experts and their collective push into high-resolution display of great value in forwarding their own strategic and business goals. New standards will be set, and working methods developed, which will define high definition imaging in the future.
- Foundations with a mandate to financially assist tax exempt groups dedicated to technical innovation in imaging, as well as those with an interest in supporting the visual arts, will find the goals of the BPS group consistent with their missions.
- A university is the natural home for this group as we will generate scientific and artistic ideas and activity at the highest level, while attracting top professionals in imaging and related fields.
- An art museum in a major city in the United States should be the venue for this first BPS display in order to publicly ground our effort in what will probably be its least understood element, namely, the intuitive nature of our quest, long understood as essential to the arts, but not usually perceived as a central to scientific inquiry.
- Individuals are the key to all the institutions outlined in the section above, whether they are visionary directors or major financial benefactors. In a quest such as ours, we will need to focus our outreach to people who, like the members of our group, will be able to see the opportunity in our project, and not view it as an unrealistic or abstract pursuit. The right partners are key to this project moving forward productively and within a reasonable time frame. Individuals who share our enthusiasm and vision will help us cut through red tape and succeed.
7. The Next Big Picture Summit
The group will convene again in June 2005.
8. BPS Founding Summiteers
Richard Benson - Dean, Yale University School of Art / MacArthur Fellow / Photography Pioneer / New Haven, CT
Red Burns - Creator & Chair, Interactive Telecommunications Program, Tisch School of the Arts, New York University /New York, NY
Carl Diegert – Lead Computational and Imaging Scientist, Digital Paleontologist, Sandia National Laboratories / former Associate Professor, School of Engineering, Cornell University / Albuquerque, NM
Bran Ferren - Co-chairman & Chief Creative Officer, Applied Minds, Inc. / former President, Walt Disney Imagineering / Los Angeles, CA
Chuck Harrison - President, Far Field Associates, Inc. / Color Scientist / Imaging and Data Security Engineer / Consultant for the Digital Cinema System Standardization Project / Seattle, WA
Michael Hawley - Director of Special Projects, Massachusetts Institute of Technology / formerly of Bell Laboratories and Lucasfilm / Cambridge, MA
Clint Hope - Executive Director for Experiential Systems, Applied Minds, Inc. / former Senior Scientist, Walt Disney Imagineering / Los Angeles, CA
Zoltan Levay - Imaging Resource Lead, Hubble Space Telescope Science Institute, Johns Hopkins University / Baltimore, MD
Tom McKnight - President, Sensorstar, Inc. / Lead Scientist for Space-Based Telecommunications and Control Systems for Deep Space and Earth Orbiting Spacecraft / Ellicott City, MD
Michael Naimark - Adjunct Professor, Interactive Telecommunications Program, Tisch School of the Arts, New York University / Adjunct Professor, University of Southern California / Media Artist and Researcher / Los Angeles, CA
Daniel O'Sullivan - Associate Professor, Interactive Telecommunications Program, Tisch School of the Arts, New York University / Developer, Quicktime VR for Apple Computer / New York, NY
David Rogers - Imaging and Visualization Scientist, Sandia National Laboratories / former Architect, former Production Software Supervisor, DreamWorks Feature Animation / Albuquerque, NM
Eric Rosenthal - President, Creative Technology, LLC. / Imaging Consultant for National Security Agency, Office of Naval Research and NASA / former Vice President of Advanced Technology Research at Walt Disney Imagineering/ Morganville, NJ
Clifford Ross – Artist-Photographer / Organizer, "The Big Picture Summit" / Inventor of the R1 High-Resolution Camera System / New York, NY
Ben Shedd - President, Shedd Productions, Inc. / Academy Award Winner / former Senior Research Scholar, Department of Computer Science, Princeton University / Boise, ID
Liron Unreich - Digital Imaging Director, Ross Studio / Photoshop Specialist / former Israeli Army Intelligence Photographic Interpreter / Filmmaker / New York, NY
 personal communication with Clifford Ross