P.J. Brown
Department of Computer Science, University of Exeter, Exeter EX4 4QF, UK
e-mail: P.J.Brown@exeter.ac.uk
ABSTRACT
This working paper tries to identify the concepts underlying the use of annotations, and also some of the related practical issues that arise when these concepts are implemented. Both paper annotation and electronic annotation are covered. Given that the latter has greater capabilities, and much greater choice of approach, a number of extra issues arise. These include how documents are stored and displayed, methods of retrieving past annotations, and the relation between annotation and editing.
We aim here to identify general mechanisms and principles behind annotation. We try to be as wide-ranging as possible. Thus we aim to cover both annotation on paper and annotation on computer screen, and we aim to cover both annotation for editing, e.g. a proof reader's marks, and annotation as commentary. Except where otherwise stated we aim to cover any kind of document, from a web page to a spreadsheet. A document may be something stored in a file, or a dynamic entity that is generated on-the-fly. We discuss, in the next Section, the nature of documents.
Annotations may be made by a single person or by a collaborative group: here we concentrate on the former as we believe that many of the principles for collaborative annotation are likely to be the same as for other types of collaborative working, and are thus well covered in the literature.
We take a wide view of what a document is. Often documents will be textual, but they may also be pictures, videos, sounds, data, etc. Documents may be constrained to follow a certain form and/or use a certain notation, and they may be structured into sub-fields. Documents may be designed for eventual viewing by humans, or they may be designed for processing by programs. Here we assume by default that documents -- and their annotations -- are for humans to see, since this is by far the most common case.
We are not greatly concerned with the nature of the documents being processed. We do, however, assume there is some mechanism for defining fragments of a document; annotations can be attached to such fragments. The exact approach to defining fragments may vary: it might be trivial, such as selecting a paragraph of a textual document, or it may be complex, such as focussing on a moving person's face within a video.
An annotation is the process whereby an author creates a document, the annotating document, and attaches it to another document, the base document. The annotating document is attached to one or more fragments of the base document: these fragments are called anchors of the annotation. In addition each annotation may have some associated metadata. Each annotation has, in principle, its own separate annotating document, though in practice a lot of annotating documents may be stored together in a single file.
Annotations may be grouped together in annotation sets, e.g. all the annotations produced by one user on a certain base document. A base document may have more than one annotation set, and, vice versa, an annotation set may apply to more than one base document -- it could even apply to every document a user loads.
In practice the annotating document is often short, even null, as when the author of an annotation on paper just underlines a passage in the base document, thus specifying the anchor of the annotation, but does not write any comment in the margin (i.e. there is a null annotating document). Indeed Marshall & Bernheim Brush found that, in a particular experiment with real annotation usage, this null case was the most common. Similarly an anchor may be null, in the sense that it is a zero-length fragment, and thus represents a position in the base document rather than part of its content. In this null case an author might, if working with paper, write a caret sign on the base document, meaning that something should be inserted in that position, and write in the margin some text to be inserted (this text is the annotating document). The caret then identifies the zero-length anchor in the base document.
In current practice, the vast majority of annotations have a single anchor. However, an example of a double anchor would be where the author circles two paragraphs (i.e. each paragraph is an anchor) and draws a line in the margin joining the two paragraphs; for this double anchor the author creates the annotating document: "these two paragraphs appear to contradict each other". In electronic annotation there is also the possibility of attaching an annotation to all occurrences of a certain word in the base document: for example all occurrences of the medical term "Goserelin" could be given the annotating document: "brand name Zoladex". Thus this annotating document could have a large number of anchors, perhaps spread over many documents.
In general, anchors of different annotations may overlap: for example one annotation's anchor may be a whole paragraph, whereas another's may be a single word within the paragraph. The simplest case where one anchor lies within another, i.e. anchors are properly nested. It is possible that anchors may overlap but not be properly nested: for example one anchor being the first and second words of a sentence, and another being the second and third words. Another example would be one annotation attached to a rectangular area within a picture, and a second annotation attached to an overlapping rectangular area. It is also possible, indeed quite likely, that an anchor is not properly nested with other entities in the base document, as happens when the anchor encompasses the last word of an italicised phrase together with the next word, which is outside the italics. These cases may be problematic as most representations in computing assume proper nesting (thus all HTML entities must be properly nested within each other). Problems would apply, for example, if annotations were embedded in a document whose mark-up assumes proper nesting. We see no silver bullet for such problems.
Ideally an annotating document should be a "first class citizen" in the sense that it can use the full repertoire of facilities found in documents; in particular an annotating document could itself be annotated.
In practice the base document is usually not owned by the author of the annotation, and the author thus cannot change the original source. Even if the author does own the base document, he may well want to keep the annotations logically separate. In this discussion, therefore, we regard annotations as conceptually separate from the base document, even though, in certain cases, they may be stored by embedding them in the base document.
Annotating a paper document is normally intrusive, i.e. the base document is defaced by the annotation and the annotations and base document are permanently tied together. (A partial exception is that the use of Post-it notes to annotate, say, a library book, provides a medium where the annotations can subsequently be removed and the book restored to its original pristine state.) In many cases the user has their own personal copy of the base document, and the intrusive nature of the annotations is a strength -- annotations and underlying document cannot be separated.
Electronic annotation systems normally offer either intrusive or non-intrusive presentation of annotations: when the base document is displayed, the user can choose whether or not to view the annotations. This choice at the display level is independent of whether annotations are intrusive at the implementation level, e.g. whether annotations are stored as embedded in the base document (see later Section "Embedded or separate electronic annotation").
Creators of information increasingly recognise the worth of metadata. Metadata is valuable for enhancing an annotation. The metadata of an annotation could give some information about its nature, e.g. whether it is a suggested insertion or deletion, or just a comment (we discuss this later, when we consider edit-annotations). With paper documents, the metadata of an annotation normally forms part of the annotating document, e.g. proof-reader's marks (metadata) are intermingled with text to be inserted; the reader is left to judge what is metadata and what is not. With electronic annotation metadata can be logically separate from the rest of the annotation, and there is a lot more scope for using metadata. Electronic metadata can record the context in which the annotation was made (e.g. author, time, location, etc.), and how the annotation is to be used (e.g. some of a reviewer's annotations may be for the author and some only for the editor). An item of metadata for an annotation may be created at the time the annotation is created and remain unchanged thereafter (e.g. date of first creation); alternatively it may change dynamically (e.g. author's current location). Some people regard annotations themselves as metadata for the base document; with this view, metadata attached to an annotation is then metametadata.
A convenient way of allowing the author to generate some aspects of metadata is to present her with a set of alternative "data types" for each annotation created. Data types might be "comment for editor", "comment for author", "correction", etc. When the annotations are subsequently read by the user, the properties of the metadata are compared with a user profile, and this determines how annotations are displayed or indeed whether they are displayed at all.
In addition, data types can be used to constrain the form of an annotation: for example if the base document consists of experimental results, then some of these results might be given an annotation of data type "provenance"; any annotating document of type "provenance" might then be constrained to consist of certain sub-fields (e.g. dates, equipment, names of people involved) -- some or all of this data might be created automatically rather than directly by humans.
In this Section, and in subsequent Sections, we concentrate on issues that apply to electronic annotations, not paper ones, though we will make the occasional reference to paper documents. We are also changing perspective in another sense: the above material was mainly about concepts, but the focus now includes more of how concepts can be implemented.
Annotations need support systems for creating them, reading them and retrieving them. For annotation of paper documents the term "support system" is a grandiose term for using a pencil: the author uses the pencil to write annotations on a base document; these can be read at the time or at any subsequent time the base document is looked at.
For electronic annotation, however, the support systems represent significant software (usually much harder to use than a pencil!). There will be an authorship system, which allows the creation and editing of annotations, and a reading system that allows people to view the annotations on a document. Both the above systems may offer, according to the user's choice, many alternative renderings of the base document and its annotations, e.g. with annotations in the margin or inserted in-line; in addition the metadata associated with annotations may be used to render different types of annotation in different ways, e.g. if the annotation represents a correction to the base document, the rendering may show the base document in its corrected form. This choice of rendering is a potential advantage over paper. The support system may also contain a sophisticated retrieval system that allows the user to make requests such as "show me all annotating documents containing the word statin", "show me all annotations created in January 2005, while I was at Southampton", or "show me all annotations where the base document had `cancer' in its title". Obviously this goes way beyond what can be achieved with paper, where the only approach is to search by eye through the annotated base documents. We discuss retrieval later.
At the moment most annotation support systems have their own way of representing annotating documents. This may be based on syntactic standards, such as XML. In addition some systems, e.g. the Gene Ontology Consortium, use RDF to add structure to XML. Nevertheless, though there is some use of standards, there are very few cases where annotations prepared on one system can be used directly on another without a messy and tedious conversion job. Thus authors of annotations generally need to assume that all readers will use one particular annotation system.
If annotations already exist for a document, the annotations may have come from one or more of the following sources:
If the author of a base document knows/requires that the reader will be using a certain annotation support system, she can exploit this in various ways by attaching annotations to the base document at the time it is created.
One example is that the author can use annotations to present features that on paper would be footnotes -- indeed it can be argued that a footnote is really an annotation but is geared to the technology of printed paper documents, and can be represented in a better way within an electronic document.
A second example is that annotations can be used to create different versions of the base document and allow the appropriate one to be presented to the reader. Assume that a document explains how to exchange an ink cartridge in a printer. Certain details of this process differ depending on whether the user has a Model A or Model B printer. The author represents Model A details by means of annotations whose metadata contain the property `Model A', and similarly for Model B. When the reader loads this document his user profile is queried to see what model his printer is. (This information may have been set automatically; alternatively it may be necessary to query the reader.) Depending on how the user profile matches the annotation's metadata, the appropriate information is then displayed. Assume this is the information for Model B. Ideally the support system will, in this case, display the Model B annotations in-line, just as if they were part of the base document, and the Model A annotations will not appear at all. In general this approach provides a promising facility for tailoring documents to users, a facility not well supported by, e.g., most current web software (CSS offers ways to do this, but many current web browsers do not offer the necessary support).
The previous Section discussed the case where the base document was prepared with the support system in mind. We now discuss the opposite case: taking an arbitrary base document, and, before presenting it to the user, preprocessing it to fit the support system, i.e. by automatically turning certain features into annotations. For example metadata attached to the base document (note that we are talking about metadata attached to the base document, not metadata forming part of existing annotations) might be turned into annotations: this metadata might supply semantic information about a phrase in the document, or provide further information about items of e-science data (e.g. provenance of experimental data or an expert's view about an area of a mammogram).
As a second example, ordinary HTML mark-up could be partially turned into annotation. For instance a document could be presented so that all material below the <H3> level (e.g an <H4> heading and the paragraphs that come under it) is presented as annotation; the main document shown to the user is only the paragraphs headed by <H1>, <H2> or <H3>. Only those readers interested in detail would follow up the annotations to show the low-level material.
Assume an electronic user has annotated a base document D, and saved the annotation set for subsequent use. These annotations can be accessed in two possible ways:
In practice both the above access methods are needed.
At this point we return more to the conceptual level, and consider an important and widely used type of annotation.
In many applications annotations are just regarded as commentary on the base document. However, in some applications, such as when someone comments on a draft paper and records their suggestions as annotations, some annotations will be of a different nature: they represent explicit (suggested) changes to the base document -- perhaps a correction of a typo or an improved wording. We call these edit-annotations, to distinguish them from ordinary annotations, which we call comment-annotations. Edit-annotations may be deletions, replacements or insertions. They may apply to any media, but here we will assume they are textual. In many applications edit-annotations are extremely common. We therefore suggest provision for edit-annotations be a built-in feature of annotation systems. Metadata might be used, as suggested earlier, to indicate the type of the edit-annotation, e.g. that it is a deletion; it could also provide a comment to explain the reason for the edit-annotation.
The extra functionality to cover edit-annotations can be provided at two levels. The first level is the display level. The annotation support system can show edit-annotations in a way that helps the reader to see what they do, e.g. a deletion can be shown as a crossing-out of the relevant text. Carrying this further, the annotation support system might, on option, show instead the effect of each edit-annotation, i.e. for a deletion the text is actually deleted, and similarly for replacements and insertions.
Secondly, the edit-annotations can be effected at the source level, i.e., if the user asks for it, the base document is actually edited using the edits specified by the edit-annotations and then saved with these edits done. (Hopefully a back-up copy is the original is kept as well.) Given this facility, the annotation support system can serve as a text-editing program. If there are comment-annotations as well as edit-annotations, then, when the edit-annotations have been effected, the comment-annotations should remain on the newly-saved document. Their positions in the document might change: e.g. if some material has been deleted before a comment-annotation, then the comment-annotation will move up the document.
Given the source level facilities described above, the way someone could make suggestions on a draft paper could proceed as follows:
This procedure is simpler and less prone to error than traditional approaches.
Following on from the example above, assume that two independent referees have each created an annotation set commenting on the same base document. This is a case where it might be desirable to merge two or more annotation sets. There are two facets of such merging:
Earlier we mentioned possible problems with proper nesting. If two independent people produce annotation sets, it is quite likely that some of them will not be properly nested. Thus one person may attach an annotation to the first and second paragraphs, whereas the other has attached an annotation to the second and third paragraphs. Therefore it is desirable that the support system deals with improper nesting.
Merging can be a sophisticated matter, as illustrated by the following example: a referee has suggested revisions to a paper and has done this by annotating the paper; the author produces a revised paper; the merging software attaches the referee's original annotations to the revised paper (a task requiring intelligence), thus allowing the referee to see whether the suggestions have been followed.
We now come to one of the most important facets of electronic annotation: the storing of annotations. Users store annotation sets for two purposes:
We assume that, to cover both of the above, all annotations are stored in a repository. Typically the repository will be part of the annotation support system. As well as storing annotations, the repository may also record further information, such as the name of the base document (though this will often be stored as part of each annotation set anyway) or the whole content of the base document. The user may have no direct awareness of the repository: behind the scenes the support system saves all annotations in its repository, and then uses this repository for later retrieval. Underneath there are many possible mechanisms for storing repositories, ranging from the trivial (a set of files in a filing system, a file-based approach) to the sophisticated (a set of interlinked individual repositories distributed all over the world). Often the user might regard the repository as if it had several levels, e.g: the user's own annotations, the annotations produced on the project the user is working on, and, at the top level, an international collaboration involving annotations. (For the higher levels the user may only have read access, not write access.) At any one time the user may set parameters to say what levels should be searched.
We next consider how annotations are stored. Most likely they will be collected together within annotation sets, rather than as single annotations. They may be stored as embedded in (a copy of) the base document -- see discussion below. Alternatively they may be stored separately from the underlying document, either (a) in a file, perhaps with a special suffix such as .ann -- this will apply in the file-based approach -- or (b) in a database within the repository.
There are design decisions concerning the storage structure of the repository. One approach is that of traditional information retrieval (IR): each IR system has a large block of storage containing all the content of the repository. This will contain documents or, more likely, copies of documents, and it may have extra information to aid fast retrieval, e.g. inverted indices. The repository content is updated periodically, either automatically, as Google does for its huge repository, or by explicit submission from the creators of documents -- in our case annotation sets will be submitted.
An alternative approach is a lighter one. The repository has no storage of its own, but, assuming a file-based approach, uses existing files instead. There can be rules for saying which files are searched as if part of the repository, e.g. all files with .ann extensions within a certain set of directories. Thus the repository simply consists of links to each of the relevant .ann that happen to exist at any one time. Obviously this lightweight approach is appealing when there are not very many annotations.
Clearly, whichever approach is used, retrieval will need to offer sophisticated searching mechanisms; these are discussed later in the Section "Searching the repository". Before that we discuss a few details concerning the repository.
Certain tasks will involve teams of people rather than individuals; even in individual tasks, the individual may change during the task, e.g. a stand-in for a radiographer who looks at medical images. Hence each document in the repository needs to have associated ownership and access rights, and there must be a mechanism for changing these.
The repository is a form of digital archive: conventional wisdom is that creating and maintaining a long-lasting digital archive is `harder than people think' (Anderson, 2005). There are potential problems of copyright -- see further (brief) discussion towards the end of this paper -- and of legacy software. If you find a ten-year-old digital document there is a high chance it will be unusable because the software needed to render it no longer exists, or exists but no longer supports the format used 10 years previously. Hence, unless a massive investment is made in a repository, it must be accepted that, over time, some of the information in it will become unusable. In most application areas this is acceptable: retrieving, say, 90% of what should be there is much better than not being able to retrieve anything.
A characteristic of a repository concerns whether it just keeps latest version or keeps all versions. As an example, assume a user (1) annotates a document and saves this in a repository, then (2) retrieves the annotations and changes them, and (3) saves the new annotations. Depending on the design of the repository, the action at (3) may or may not erase the annotations previously saved at step (1). Keeping latest versions is probably adequate for most applications.
When the user creates an annotation his needs will relate to the task he is doing. Sample tasks are: correcting a manuscript; reviewing papers for a conference; conducting research in legal processes; producing a reference work such as `The History of Jigsaws'; searching the current literature for all evidence on whether a certain drug affects blood clotting; providing a compendium of business practices within a company; comparing two ancient documents; examining a medical image for signs of disease. Some of these tasks are ephemeral, whereas others may last nearly a lifetime. In some cases even after the task has been completed, permanent records still need to be kept, for example as a defence against potential lawsuits many years in the future. Indeed in many research tasks annotation of technical papers, data, etc., may represent a significant part of the investment in the task. Some tasks may involve creating tens of thousands of annotations. For all but the most ephemeral tasks there is a potential need to look at annotations again long after they have been created. This re-looking may involve going straight to an annotation that the user remembered he created, say, two years previously. More often, however, it will involve a search of past annotations to find the required ones. Probably the most important feature of an electronic annotation system is to provide a mechanism to aid such searches. This will involve creating the repository of annotations, and a way of retrieving from the repository.
Sometimes the retrieval need will be a straightforward search, like `get me all annotated documents with property XXX'. Often, however, the search will be more hit-and-miss: the user wants to retrieve a certain annotation made in the past, but only remembers sketchy details about it: the rough creation date and where he was located at the time, some words of its content, and perhaps some detail about the content of the base document. To help such searches the retrieval system must allow searching using as many independent criteria as possible. Some examples are:
Often finding an annotation will be a needle-in-a-haystack problem. Allowing the search to seek the target from different perspectives, e.g. start with context and then look at content, is one way to make finding the needle more practicable.
Marshall has found that people which write annotations on paper are typically careless in specifying anchors. For example they might intend to specify the phrase "double standards" as an anchor, and they circle it or underline it; because of carelessness they actually specify "ouble standard". This does not matter on paper -- which is doubtless why it happens so often. If electronic annotation supports retrieval according to the content of anchors, carelessness does matter. In the above example, if the annotation was stored in the repository, a search for "double" would fail. Part of the answer may lie in automatic "rounding" whereby anchors are always attached to whole words.
It has also been found (Marshall & Bernheim Brush, 2004) that when people write annotations on paper documents the annotation itself is often written carelessly. One possible problem is legibility. This problem does not occur with electronic annotation if the annotations are typed at a keyboard. Leaving this aside, even if an annotation can be read, an author, on seeing it some time later, often has trouble understanding what was meant. Certainly if the annotation if not shown in its context in the base document, it will usually be impossible to understand. This has implications for retrieval and for displaying the results of retrieval.
Having discussed the repository, we now move on to some issues, mostly practical, about the base document.
The authorship system needs to present the base document to the user, so that the user can specify anchors and attach annotating documents, metadata, etc. With electronic annotation there is a choice of level at which the base document can be viewed. The lowest level is to render the base document in the form an end-user views it (e.g. from a PDF file), thus creating an image, and allow the user to annotate this image. Annotations are attached to a position in the image. This lowest level roughly corresponds to the way paper documents are annotated. The approach loses most of the advantages of electronic annotation (unless positions of the image can be related back to the source of the base document). It may also require the base document to be rendered in exactly the same way each time a reader looks at it, and, if annotations are shared, in the same way for each reader. Furthermore it is inflexible to change.
The ideal is that all annotations are attached to the source of the base document, and are therefore at the same logical level. Thus the annotator should see the base document in the same way that its author does. For a web document this would allow annotation to the mark-up of a document, not just its textual content.
In most annotation systems the original author of the base document is unaware that other people have made annotations to her document. In some limited user groups, however, who share the same repository, there is scope for providing such an awareness. Then an author might ask to see all the annotations made on her document by members of the group. The extreme case of a limited group is a group of one, i.e. awareness of the author's annotations on her own work; it should be easy for a repository to provide facilities to support such awareness.
Assume that a base document has some existing annotation sets attached to it, and the creators of these want their annotation sets to be carried over to any future versions of the base document. If the author of the base document has no awareness, then, if the base document is subsequently edited, there are likely to be problems with these annotations. Indeed some of them might become unusable.
If, however, there is awareness, extra features can be added to the support system to allow trouble-free editing of base documents. The key extra feature is as follows: we have already said that the support system should allow the editing of annotations; the extra feature -- an natural extension of this -- is to allow editing of the base document too.
With this feature, editing of the base document can proceed as follows:
Some editing systems create comprehensive histories, and these can be invaluable for recovering from disasters. Obviously this is desirable in the above case too.
We have suggested above that the support system be capable of effecting edit-annotations; this provides a facility for editing the base document. Do we still need a separate facility for directly editing the base document? The answer is almost certainly yes, as creating an annotation just to do an edit would be cumbersome.
When a user creates some electronic annotation that he might want to use later he asks for it be stored. Normally this is done by the support system. There is a choice of how the annotation is stored; the options are:
If one of the above options is chosen, and, later on, it is desired to switch to the other, this is not usually a problem. It is generally easy to perform a mechanical conversion either way between (1) and (2), provided, of course, that in case (1) the base document is still accessible (e.g. it is not a URL to a page that no longer exists). In fact in many cases the user will not know or care whether their support system uses (1) or (2). There are, however, cases where it may matter.
For embedded annotation there will be a single document stored, which (we assume) the user can access. Hence the user has some control. On the other hand if annotations are separate they might be stored in some esoteric form unique to the support system, and hence the user is totally dependent on the support system to get information out. In the embedded case, users can publish documents with annotations just by, e.g., putting them on the web. Moreover the documents can be made available to web crawlers so that search engines will then allow the world to find the documents -- a big advantage to authors who want to air their work. Any reader who accesses an annotated document will need to have the same support system -- otherwise they will probably see the base document without annotations. A possible downside, on the other hand, is that readers will be able to see all the annotations in an annotation set (if necessary by examining the source of the published document): there is therefore no scope for keeping some annotations secret, such as reviewer's comments not meant for the author.
Overall embedding has some attractions, and certainly might be used as in applications where annotations are to be published to the world.
We now move to a more fundamental issue one that affects all software: change.
In the paper world, each annotation is made explicitly by the author and does not change unless the author changes the annotation. Moreover the base document does not change, apart, perhaps, from turning yellower with age. Essentially the paper world is a static world.
In the electronic world there is scope for much more dynamic behaviour. We here discuss three possible aspects of this. The first aspect concerns the base document. If, say, the user annotates a web page, then two approaches are possible:
In practice some applications need to work with dynamic base documents, and for others static base documents are ideal. We discussed dynamic base documents and their editing earlier, when we considered awareness.
A second aspect of dynamic behaviour in the electronic world is that when annotations are partly or wholely generated by the computer, rather than by the user, then this process may be -- and usually is -- dynamic. Our previous example of annotating all occurrences of "Goserelin" is an example of this. Such annotation is likely to be dynamically produced: if the base document has changed since it was last read, and contains two further instances of "Goserelin", then these two extra occurrences will automatically be annotated too. Thus we have dynamic annotation. The work of Rhodes and Maes carries this further: when an base document is loaded a whole set of annotations, tailored especially to the user's current interests, is created on-the-fly and applied to the base document.
A third aspect is that electronic documents may themselves have dynamic behaviour, such as, within web pages, hyperlinks or the effects of Java scripts. There might be flashing/changing panels, "tours" where several documents are visited in turn, etc. This applies both to base documents and annotating documents. In an extreme case a base "document" could be a series of actions such as selecting menu items. A user may for example attach the annotating document `Changing the default font' to a set of menu actions selected by the user; these actions form the base "document" of the annotation, and selecting the annotation would cause these actions to be played back (like a form of macro).
As an alternative example of this third aspect, the purpose of an annotation that represents a correction to a document may be to change a place name, currently in plain text, to a hyperlink to the web site of the place. Here the annotation does not affect the textual content of the base document but changes its dynamic behaviour.
As can be seen change is an issue that affects annotation systems both at a fundamental and a detailed level.
Dynamic behaviour of documents is only one facet of change; over time there are likely to be more far-reaching changes in the computing environment. Over any length of time there will also be changes in metadata (formats, data types, usage of terms, etc.), changes in users' habits and environment, changes in organisation, etc.
There is no silver bullet (see, however, Phelps & Wilensky for robust URLs). Careful and flexible design will reduce some of the problems, but will not eliminate them. Storing of contextual information also helps, e.g. storing what the world is like (notations used, parameter values, current versions of software, etc.) at frequent intervals -- or whenever there is change -- can be a great aid.
In the last part of this paper we try to `mop up' a number of issues, mostly general ones, that impinge on annotation -- in some applications they impinge in a major way.
As we said at the start of this paper, we are not here interested in collaborative annotation in the sense of a group of people simultaneously annotating a document. Obviously, however, annotations need to be exchanged among colleagues and perhaps even published. This can be done in the same way as people share and publish any document; e.g. through e-mail, file permissions (for local colleagues), and web sites. A good underlying principle is that at one time just one person holds the baton in the sense they own a particular annotation set and can therefore change it.
In many environments authors have constraints on the form of document they can write. This may take the form of a house style or, more strictly, a template that every document must follow. Exactly the same constraints may apply to annotations. One example is the GandrKB project, which is concerned with putting annotations on descriptions of genes; large numbers of such annotations may be produced. To make the collection of annotations more valuable (e.g. more easy to search for certain properties), authors are constrained on the form of annotations they can write. Often such annotations will be read by programs, not by people; programs are not good at reading free-form information, but prefer standard formats, standard usage of terms (ontologies), etc. As a second example of constraints, proof readers are usually severely constrained on the forms of annotation (e.g. proof marks) they can write.
In the electronic world it is possible for authorship systems for annotations to impose the required constraints, so that only "correct" annotations can be produced.
Making annotations on paper is easy and convenient. In comparison annotating an electronic document will always be slow and clumsy, notwithstanding the good user interfaces provided by systems such as iMarkup. The compensating advantages come from the extra facilities offered by the electronic world, notably retrieval of annotations, as discussed above.
The long-term solution to these usability problems is to combine the paper and electronic worlds so that to the user they are one: advances in cameras, projection, and OCR offer the possibility for a user to work in a paper world yet have full electronic capture and retrieval. Although pilot systems such as the Digital Desk were available in 1993, such hardware has been slow to become widely used. Let us hope its time will soon come. Products based on Anoto pen technology are one promising avenue. A less ambitious solution is the use of electronic devices that provide an interface closer to pencil-and-paper. Tablets are an example of this.
If we consider wider aspects of usability in the electronic world, it is obvious that all aspects of any annotation system, especially authorship and retrieval, need to be made as usable as possible -- with minimal extra user effort. If contextual information needs to be captured it is highly desirable that this be done automatically. Users are notoriously reluctant to fill in electronic forms to give their preferences, location, etc.
A further, and especially challenging, aspect of usability is that the annotation system must work seamlessly with other software the user employs to create or read documents: browsers, word processors, picture editors, etc.
It is possible that different annotation applications will require different interfaces. In particular in some applications, such as those involving ancient documents, annotations may be extremely dense, e.g. an annotation on every word to explain its meaning. In other applications annotations are relatively sparse and might need highlighting to help the user find them. Following on from this, there may be a need for different interfaces for macro-annotation (annotation attached to a whole document or to a big part of it) and micro-annotation (annotation attached to a single paragraph or something smaller).
The woolliness of this Section indicates that the issues require much more thought!
A characteristic of really successful software is that it finds uses well beyond its original purpose. The World Wide Web is a prime example of this.
Although such uses are, by their nature, impossible to predict, we will give some pointers to where extra uses of annotation systems may lie. Firstly, if an annotation system offers easy capture and effective retrieval, it can be used as a general-purpose memory bank. If the user wants to remember a document he is reading he just marks it in simple way; the mark is treated as an annotation and the document is saved in the repository. The retrieval mechanisms of the repository can then be exploited.
Secondly there is scope for combining annotation with other document-manipulation facilities such as hypertext, bookmarks, and history lists. One general system might be able to combine the advantages of all of these. Indeed the similarities between hypertext linking and annotation have already been explored (Brown & Brown). More radically Nelson's ideas of transliterature offer huge possibilities.
In addition to the above, there are potential applications that use annotations as a "free" body of evidence about the base document. The work of Mizzaro is relevant here. Just as a technical paper can be rated by the number of later papers that cite it, so the nature of annotations attached to a paper can provide evidence. Mizzaro is also concerned with rating readers: do a certain reader's opinions have authority? Again the nature of the reader's annotations provide some evidence about this, especially if annotations have data types indicating the nature of the annotation.
A parallel to this, based in the world of paper documents, has been pointed out by Dr. David Shaw. To quote: "In the humanities the idea of chronological layers of variant text is found in the field of textual transmission and textual editing of classical and medieval texts. ... A recent new field of research in the history of the book is the study of activities of former owners in leaving marks on them ... ; there are potentials here for electronic annotation on digital versions of early printed books [thus capturing the marks added to the original paper form].".
We strongly support the view that each software component should concentrate on doing one task and doing it well. It should not try to do other, related, tasks, but instead should provide a way for it to be used as a sub-system by applications that concentrate on these related tasks. This view was made famous as an original design principle of UNIX.
Thus we believe that an annotation system should just be designed for annotation, but, given that annotation is a useful technology in many applications, should allow itself to be used as a sub-system by the applications -- hopefully seamlessly as far as the end-user is concerned.
One example of such an application is discussion groups, especially those centred on a single base document such as a literary work. In these discussion groups, a matter for discussion can be written as an annotation anchored to the appropriate fragment of the base document. A later posting might reply to one of these annotations, i.e. be anchored to all or part of the original annotation, and there may subsequently be replies to replies. Clearly several different authors will be involved. There need to be restrictions on changing or deleting original annotations if they have replies to them. Indeed there may be a rule that every posting is write-once. Our view is that the way to implement such functionality -- which is primarily discussion group technology -- is via a discussion group application. This application would call an annotation system for the basic functionality of anchoring one document to a fragment of another.
A second example is parallel documents, e.g. a document in German and a corresponding English translation. The application supporting this may allow annotation and have a rule that the annotations, like the base documents, must be kept in parallel versions. The application would enforce its own rules but call an annotation system to do the basic annotation task.
Issues of copyright, licensing, charging and provenance can apply to any document. Hence they may apply to either the annotating document or the base document or to both. In particular, licensing issues may be a bar to saving a base document permanently in a repository and/or to making a repository publicly available.
These issues are clearly highly important, but we do not currently believe that the practice of annotation raises any new aspects over and above those already covered widely in the literature. Therefore we have not discussed these issues here.
The concepts of annotation are relatively simple. However, if we look beyond the concepts, and examine practical matters about the environment in which the concepts will apply, we find that, particularly in the electronic world, annotation impinges on a host of other issues. These range from intellectual property to optimal techniques for searching. We have tried to cover these various issues, albeit mainly in outline, though the result is a somewhat rambling dialogue. Hopefully, however, bringing the information together in one place will be an aid to understanding the field, and lead the way to future refinement and simplification.