Experimental Interfaces Involving Visual Grouping During Browsing

Dr Stan Ruecker
Assistant Professor
Humanities Computing Program
Department of English and Film Studies
3-5 Humanities Centre
University of Alberta
Edmonton AB  T6G 2E5

This paper is based on a conference presentation given by the author at Access 2005, held in Edmonton, Alberta on October 17-19th, 2005.

Author Biography

Dr. Stan Ruecker is an Assistant Professor of Humanities Computing in the Department of English and Film Studies at the University of Alberta. His PhD research was on the affordances of prospect for computer interfaces to large, interpretively-tagged text collections. His current research interests are in the areas of computer-human interfaces, text visualization, and information design.


This paper provides a brief overview of a number of experimental interface design projects being carried out collaboratively by teams of researchers at the University of Alberta and elsewhere. One goal of this interface research is to explore the principles of rich-prospect browsing interfaces, which I have defined (Ruecker 1) as those where some meaningful representation of every item in a collection is combined with tools for manipulating the display. Often this manipulation is for the purpose of carrying out some portion of a research task: the interfaces lend themselves to exploratory and synthetic activities, such as knowledge discovery and hypothesis formulation. The projects summarized here begin with a browsing prototype originally designed for the task of pill identification (Given et al.). This prototype was subsequently extended into a prototype for browsing conference delegates and other groups of people (Ruecker et al.).  Another direction is represented by a nuanced system based on the mandala (Cheypesh et al.) intended for examining any collection that has been encoded with an XML schema. The Mandala Browser uses combinations of “magnetic axes” selected by the user from the available tags. Next is the set of specialized interfaces for the Orlando Project (Orlando Team), intended to provide a set of discrete entry points into the deeply-encoded electronic history of women’s writing in the British Isles. Our project on tabular interfaces provides a variety of spaces designed to assist the user in using thesauri for multilingual query enhancement (Anvik et al.). The final project described below is NORA (Unsworth), which relies on the power of the D2K data-mining tools at the National Centre for Supercomputing Applications at the University of Illinois at Urbana-Champaign. The goal of NORA is to give humanities scholars a workspace for exploring the system-identified features of common documents and further documents that have been recommended by the system. Each of these projects is discussed within the framework of visualizations involving browsing through dynamic grouping.


Computer-Human Interaction, Interface Design, Rich-Prospect Browsing, Multilingual Query Enhancement, Data Mining.


The goal of this article is to summarize recent work at the University of Alberta and collaborating institutions involving the design of prototype interfaces for browsing collections and individual documents. We are hoping through these projects to investigate and elaborate on our understanding of what I call rich-prospect browsing (Ruecker 1), where a meaningful representation of each item in a collection serves as the basis for a set of tools for organizing those representations in useful ways. The principles behind rich-prospect browsing interfaces were developed to reflect ideas that have emerged over several decades of related research by interface designers, information designers, landscape theorists, and ecological psychologists.

For instance, the benefits of a visualization approach that emphasizes a prospect view involving the individual representation of collection items is nicely summarized by Tufte in his discussion of what he calls “small multiples”:

Small multiples, whether tabular or pictorial, move to the heart of visual reasoning–to see, distinguish, choose…. Their multiplied smallness enforces local comparisons within our eyespan, relying on an active eye to select and make contrasts rather than on bygone memories of images scattered over pages and pages. (p. 33)

Previous projects employing similar strategies include Bederson’s Photomesa, which showed a display of digital images that could be dynamically categorized by the user, Harris’s “10 by 10,” which creates an hourly array of 100 images drawn from online news sources, and Turner’s tag browser, which uses the Flickr open Application Programming Interface (API) to provide a means of accessing the millions of online Flickr photos by visually browsing their folksonomic tagging.

For collections that contain only a small set of items, numbering in the thousands or fewer, it is possible to consider the design of interfaces where the default display shows every item in the collection. By showing the entire collection in some form in one screen, such interfaces shift the user’s cognitive task from retrieval to browsing. For the designer of such interfaces, the task is to provide a choice of meaningful representations and a range of tools that will be useful in manipulating the representations. The tools might provide functions such as grouping, sorting, subsetting, magnifying, selecting, graphing, charting, or visualizing to help the user obtain insight into what is available and how the items in the collection relate to one another.

For larger collections, that number in the millions of items (for example, many library systems), or billions of items (the internet), rich-prospect interface methods often require an initial subsetting of the collection, whether through retrieval or other means (such as navigating a hierarchy). However, once the set available to the user has been reduced to a few hundred or even a few thousand items, there again becomes available the variety of strategies that can be selectively applied to enable the user to dynamically interact with the subset in various meaningful ways.

From a theoretical perspective, the exploration of online browsing environments can be situated within the design of new digital affordances, which is to say new opportunities for action in the interface (Gibson, Vicente). As Frascara (xvii) points out, such affordances are particularly attractive when they exist in a context of an environment specifically intended to support and extend communication: “We need so much to see what surrounds us that the sheer fact of seeing a wide panorama gives us pleasure.” The design of a successful environment for communication in turn rests on the ability of the system to support and extend current information-seeking behaviours into a digital browsing context, which requires an understanding of those behaviours and the people who engage in them.

The five principles of rich-prospect browsing interfaces have been defined as follows:

While developing these principles, we have simultaneously been involved in a variety of research projects that are attempting to apply them to different kinds of tasks, data and a range of users. Each project goes through a series of stages, beginning with either static or kinetic sketches that can be used to discuss the theories or principles involved, followed by working prototypes, and culminating in production versions that can be widely used online. It is possible to carry out different forms of user study at each stage.

Pill Identification

One of our first rich-prospect browsing prototypes (Figure 1) was designed for the task of pill identification (Given et al.). It combines a thousand photos of pills with tools for grouping them by colour and shape. The user can choose either scheme to begin the grouping, and can continue by creating subsets of either the entire display or some portion thereof. There are also magnification tools that allow the image of each pill to be scaled until it almost fills the entire screen. Clicking on the image of a pill calls up the information about the pill in the panel on the lefthand side.

We carried out a small qualitative study of a dozen seniors interested in online health information. Their task was to identify three pills using the retrieval function at www.drugs.com, and to identify another three pills using the browsing strategy in our prototype. Some of the participants identified a preference for being able to try retrieval within browsing. They also expressed appreciation for being able to see all the items, although they had difficulty with the small scale of the initial images, in part because the “zoom” tool was not easily recognized as a magnification device.

Figure 1. The Pill Identification Project. Users were able to visually group photos of pills in order to identify the pill in hand. We learned, among other things, that each country has its own pills, and that the word “zoom” is not synonymous with “magnify” for many seniors. There is currently no online version of this prototype. Images and data courtesy of www.drugs.com.

Delegate Browsing

One of the related concepts we are exploring is how design research can be informed by design transferability (Chow and Ruecker), where researchers develop a prototype to explore ideas in one domain, then shift the design to a new domain with a new kind of user. For example, the pill identification system was subsequently transferred into a prototype (Figure 2) for browsing conference delegates and other groups of people (Ruecker et al.). In this case, the design transfer occurred at the theoretical level, since the underlying code of the prototype was completely rewritten, so that the Delegate Browser prototype could be made available online.

While the pill identification system provided only a rudimentary set of tools for managing the images, the delegate browser extends the kinds of sorting that are possible by combining roughly a dozen variables relating to the person’s professional life and visual appearance. Using the delegate browser, it is therefore possible to group one’s colleagues at a conference by features such as hair colour and the presence or absence of eyeglasses, in conjunction with other items such as keywords from their paper and their institutional affiliation.

We are in the process of carrying out our first conference trial of the system, in conjunction with the annual meeting of the Text Encoding Initiative (TEI) in Oct 2006 in Victoria.

Figure 2. The Delegate Browser. This interface transfers the idea of image sorting into the area of social capital management for academics attending conferences. It is somewhat more complex than the pill interface, since there are more than a dozen criteria for grouping and sub-grouping the images of the delegates. Online demo at www.digital-profiles.com.

Mandalas as Interfaces

The mandala browser (Figures 3 and 4) is a nuanced browsing system intended for examining any collection that has been encoded with an XML schema, using combinations of attracting “axes” selected by the user from the available tags  (Cheypesh et al.).

To use the mandala, the user opens an XML document and defines for the system the tags that are relevant for browsing. These tags form a list of items that can each be individually attached to an axis that appears in the central space.

Each axis serves to draw items in the collection from the periphery of the mandala into the centre. Two axes acting in concert will each draw items from the periphery, with some of those items being shared in the space between the axes. If the user adds a third axis, the items shift appropriately into the new space between the three points, and so on, for as many axes as can physically fit within the mandala.

Figure 3. The Mandala Browser. This online system allows users to carry out nuanced investigations of entire collections or individual documents, based on their XML encoding. This screen shot and the next show the user browsing speeches in the First Act of the play Romeo and Juliet. In this first image, the speeches of the play appear as dots placed randomly around the periphery, but the first (blue) axis has not yet been set up.

Figure 4. The Mandala Browser with Axes Defined. In this second image, the blue items (right-hand side) are speeches by Romeo, the green items (lower left) are speeches by Juliet, and the red items (upper left) are places where anyone in the Act uses the word “war.” An online demo is available at mandala.humviz.org.

The Orlando Project

The Orlando Project (Figure 5) is a history of women’s writing in the British Isles, developed by research teams at the University of Alberta and the University of Guelph  (Orlando Team). It has recently been published by Cambridge University Press. It includes a set of specialized interfaces intended to provide discrete entry points into documents about women’s lives, their writing careers, and historical events. By drawing together elements selected from the various documents, Orlando allows the user to dynamically assemble information on a vast number of topics.

Although not technically a rich-prospect interface, since it does not provide a default display of every item in the collection, it expresses some of the other principles of rich-prospect browsing by giving the user insights into the tagging, combined with tools for sorting and organizing not only the documents themselves, but also selected portions of the documents.

Figure 5. The Orlando Project. The Tag Search page shows one of three entry points in the panel on the left; allowing access through people, chronologies, and the rich encoding. This search is set up to look for information about British women writers in the Restoration and 18th Century who experienced penalties of some kind related to their writing. Orlando is available by subscription through Cambridge University Press.

Multilingual Query Enhancement Using Thesauri

This project (Figure 6) provides a variety of spaces within a tabular arrangement designed to assist someone in using thesauri for multilingual query enhancement (Anvik et al.). In this instance we drew upon previous work by Bertin on using tables to experiment with data, and on our own previous project dealing with tables as interfaces (Ruecker and Liepert).

The resulting interface places the terms from the Government of Canada Core Subject Thesaurus (Core) into a table that indicates their relationship to the current query term, whether related, narrower, broader, preferred or non-preferred. Core contains approximately 4000 terms in both English and French. The user has the choice of writing a query in this system in either language, then automatically generating a multilingual query that is roughly equivalent, based on the thesaurus vocabulary.

Figure 6. The Multilingual Thesaurus Interface. This sketch shows the three spaces that constitute our experimental tabular interface for enhancing multilingual queries. An online prototype is in development but not yet available.

NORA: Data Mining for Humanities Scholars

The final project in this survey is NORA (Figure 7) which stands for “No One Remembers Acronyms” (Unsworth). It relies on the power of the D2K data-mining tools at the National Centre for Supercomputing Applications at the University of Illinois at Urbana-Champaign to give humanities scholars a workspace for exploring documents. The users identify good and bad examples of whatever material they have in mind, and the system attempts to recognize features that represent that material. These features can be examined by the user or used by the system to recover additional documents that closely resemble the ones specified by the user.

One of the early demos of the system involved literary experts who specialize in the poetry of Emily Dickinson looking at a collection of her poems and rating them according to their erotic content (Plaisant et al.). Once the system had lists of equal numbers of positive and negative examples (in this case, of ten or so poems that the experts had rated “hot” and “not hot”), it could identify features common to the poems in both sets, and use those to suggest other poems that might fall into either category. The experts could then analyze the features the system identified as well as the lists of suggested poems.

Figure 7. The NORA Oil and Water Interface. The NORA project is attempting to make data mining accessible to humanities scholars. This image shows the automated division of the collection into three sets, with the background items being neutral, and the two circles representing those items that the system recognizes as resembling those that the user has identified as good and bad examples of the subject under investigation. An online demo using an earlier concept for the interface is at www.noraproject.org.

Conclusions and Future Research

One of the purposes for the interface projects described in this paper was to provide the basis for exploring the concept of rich-prospect browsing, where some meaningful representation of every item in a collection is combined with tools for manipulating the display. We have worked so far with a range of collections that vary in terms of content, encoding, and the extent to which underlying algorithmic processing is available. Several of the projects use default representations that are image-based (such as photos of pills or people’s faces), while others rely primarily on text representation (such as author name). Our early results suggest that, perhaps contrary to expectations, people are generally not overwhelmed by large displays of items, provided that attention has been paid to the details of the design, and tools are available for working with the display by reorganizing the items, whether through grouping, sorting, or subsetting.  We plan to continue our research by expanding the toolsets currently available to take better advantage of the opportunities emergent from complex data and processes. We are also planning experiments which examine different ways of changing one form of meaningful representation for another as part of the browsing process. Our future directions are set in part by these theoretical interests and in part by the possibilities that arise in connection with new data sources and improvements in online technologies. Each of our projects allows us to carry out different forms of user study, adding to our overall understanding of both the potential and the limitations inherent in rich-prospect browsing and enabling us to create better and more useful interfaces to digital collections.


The author wishes to acknowledge the support of the Killam Trust, the Winspear Foundation, the Andrew W. Mellon Foundation, the Social Sciences and Humanities Research Council of Canada, and the Natural Sciences and Engineering Council of Canada.

Works Cited

Anvik, Karl, Ali Shiri, Ximena Rossello, and Stan Ruecker. “A Tabular Thesaurus Browser to Support Multilingual Queries.” Paper delivered at the Society for Digital Humanities (SDH/SEMI) Conference, York University, Toronto. May 29-31 2006.

Bederson, Ben B. “PhotoMesa: A Zoomable Image Browser Using Quantum Treemaps and Bubblemaps.” Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, Nov. 11-14, 2001, Orlando, Florida. New York: ACM Press, 2001. 71-80. The ACM Portal. 16 Nov. 2006. <http://doi.acm.org/10.1145/502348.502359>.

Bertin, Jacques. “Matrix Theory of Graphics.” Information Design Journal. 10.1, (2000/2001): 5-19.

Cheypesh, Oksana, Constanza Pacher, Sandra Gabriele, Stéfan Sinclair, Drew Paulin and Stan Ruecker. “Centering the Mind and Calming the Heart: Mandalas as Interfaces.” Paper delivered at the Society for Digital Humanities (SDH/SEMI) Conference, York University, Toronto. May 29-31 2006.

Chow, Rosan and Stan Ruecker. “Transferability – A Wonder on the Ground of Design Research.” Paper delivered at the Wonderground Conference, Lisbon, Portugal. Nov. 1-5 2006.

Frascara, Jorge. “Introduction: Creating Communicational Spaces.” Designing Effective Communications: Creating Contexts for Clarity and Meaning.  Ed. Jorge Frascara. NY: Allworth Press, 2006. xiii-xxi.

Gibson, James J. The Ecological Approach to Visual Perception. Boston: Houghton-Mifflin, 1979.

Given, Lisa, Bess Sadler, Stan Ruecker, and Sandra Gabriele. “Similarity Clustering as a Design Principle for Digital Collections: Creating Usable Interfaces for Seniors.” Paper delivered at the 2005 Conference of the American Society for Information Science and Technology (ASIS&T), Charlotte, North Carolina. Oct 28 - Nov 2 2005.

Harris, Jonathan. 10 by 10. 2004. 20 Sept. 2006.  <http://www.tenbyten.org/10x10.html>.

Orlando Team. “Between Markup and Delivery; or Tomorrow's Electronic Text Today.” Mind Technologies: Humanities Computing and the Canadian Academic Community. Eds. Ray Siemens and David Moorman. Calgary: University of Calgary Press, 2006. 15-31.

Plaisant, Catherine, James Rose, Bei Yu, Loretta Auvil, Matthew G. Kirschenbaum, Martha Nell Smith, Tanya Clement, and Greg Lord. “Exploring Erotics in Emily Dickinson's Correspondence with Text Mining and Visual Interfaces.” Paper delivered at the Joint Conference on Digital Libraries (JCDL), Chapel Hill, North Carolina. June 11-15 2006.

Ruecker, Stan. “Affordances of Prospect for Academic Users of Interpretively-tagged Text Collections.”Diss. University of Alberta, 2003.

Ruecker, Stan, Michael Lewcio, Michael Plouffe, and Maryanne Wynne. “I Never Forget a Face: A Rich-prospect Image Browser for Conferences.” Paper delivered at the Society for Digital Humanities (SDH/SEMI) Conference, York University, Toronto. May 29-31 2006.

Ruecker, Stan and Susan Liepert. “Strategies for Creating Rich Prospect in Interfaces.” Paper delivered at the Joint International Conference of the Association for Literary and Linguistic Computing (ALLC) and the Association for Computers and the Humanities (ACH), Göteborg, Sweden. June 11-16 2004.

Tufte, E.R. Envisioning Information. Cheshire, Connecticut: Graphics Press LLC, 1990.

Turner, F. Flickr Related Tag Browser v1.1. 2006.  10 March 2006. <http://www.airtightinteractive.com/projects/related_tag_browser/app/>.

Unsworth, John. “Forms of Attention: Digital Humanities Beyond Representation.” Paper delivered at CaSTA 2004: The Face of Text, the 3rd conference of the Canadian Symposium on Text Analysis, McMaster University, Hamilton, Ontario. November 19-21 2004.

Vicente, Kim J. “Ecological Interface Design: Progress and Challenges.” Human Factors. 44.1, (2002): 62-78.