Electronically Integrating the Reading and Writing of
Documents: an Unexploited Aid to Education
P.J. Brown, Department of Computer Science,
University of Exeter, UK,
P.J.Brown@ex.ac.uk
H. Brown, Department of Computer Science,
University of Exeter, UK, H.Brown@ex.ac.uk
Abstract: Current computer tools are normally designed either to aid reading, such as browsers, or to aid writing, such as word processors or editors. Here we explore ideas of electronically integrating reading and writing, which could bring benefits to all in education — where reading and writing are such vital activities. A prime mechanism for achieving the integration is a generalised form of annotation.
Reading and writing are two of the basic skills gained in education. Students, as they proceed through their education, also acquire skills in integrating reading and writing. The most important of these skills is the ability to annotate: while reading a document, the student writes notes on that document; each note, which may be textual or graphical, is attached to a small portion or fragment of the document content. The fragment, which is typically a few words or a section, acts as the anchor of the note. Attaching a note in this way is an example of writing-while-reading. Another skill is, when writing new work, to read existing work and to use this material in the new work. At its extreme this becomes plagiarism, but in normal use it captures a basic need: building on what has gone before. Sometimes the reading of other work is done it advance, but often it is interspersed with writing — a case of reading-while-writing. If a student becomes a university undergraduate and perhaps later a postgraduate, skills of annotation and of using what has already been written become increasingly vital. Successful students, therefore, must be good at integrating reading and writing. At a more fundamental level, several educationalists (e.g. Bartholomae & Petrosky, 1986) believe that the two skills should be taught in tandem.
Many aspects of integrating reading and writing are trivial if the student is working with paper. Thus annotating a paper document while reading it simply consists of scribbling notes in the margin, or just underlining relevant passages. Such activity barely interrupts the flow of reading. Any paper document can easily be annotated, though of course if the document is read-only — as a library book should be — a copy must be made first. Reading-while-writing is not quite so well supported in the paper world. Perhaps, while they are writing, the student might have several relevant books open on the desk, but everything is separate.
In the world of on-line documents rather than paper documents, integration of reading and writing is poorly supported — in spite of its advantages being highlighted by visionaries such as Bush (Bush, 1945). For a start there is a division between software tools aimed primarily at reading, such as web browsers, and software designed primarily for writing, such as word processors, editors, formatters, and web page creation tools. The reasons for this division are historical, but nevertheless the division has persisted. A consequence of this is that the electronic world is often far worse than the paper world for performing tasks that need integration. This particularly applies to annotation. Annotation software exists, but usually it is tied to only one sort of document, such as a web page (Kahan et al, 2001; iMarkup, 2003), and using such software does not come close to matching the ease-of-use of paper annotation. The title of a paper by (Ronchetti, 2002) says it all: Why web page annotation tools are not killer applications.
In terms of reading-while-writing, the electronic world does a bit better. At the low level, it is easy to do an electronic cut-and-paste to place an extract from an existing document into a currently-being-written one. Moreover cut-and-paste can work for multi-media material. At a higher level, although the field of view on a screen is usually less than a physical desktop, it is possible to have several documents open at once, and to use electronic methods to search these and find relevant passages.
Our vision (Brown & Brown, 2004), and the theme of this paper, is that it should be possible to make the electronic world better rather than worse than the paper world for integrating reading and writing. This paper identifies opportunities and threats to achieving this. In total it identifies seven opportunities and three threats. The paper is essentially an ideas paper, though we make brief reference to some practical work we have done. We believe that implementing even part of our vision would be an aid to all students.
The key to realising this vision is not just to create electronic facilities that imitate the paper world, but to provide facilities that go far beyond what paper can ever achieve. A flavour of how this can be done is provided by the pioneering work of Rhodes and Maes at MIT (Rhodes & Maes, 2000; Rhodes, 2000), who have created the Remembrance Agent and Margin Notes systems, and have made them available on trial to MIT students. A Remembrance Agent is designed to aid reading-while-writing by pro-actively suggesting documents that are relevant to what the user is currently writing. As a first example, if the writer has just written a paragraph, the Remembrance Agent may suggest one or more documents the user might read, as they appear to cover the same material as the paragraph. As a second example, if the user has just typed a person’s name, the Remembrance Agent might bring up a link to the most recent e-mail received from that person. Clearly such pro-active behaviour can become an annoying distraction to the writer rather than a help. To try to combat this the Remembrance Agent provides a discreet interface, and plenty of user controls. Interestingly, the mechanism used by the Agent to present suggestions is pro-active annotations; thus in the first above example the Agent would annotate the paragraph just written with the name of a paper the user might want to read. Clicking on this annotation will lead, via increasing levels of detail, to the content of the suggested paper. In contrast with our previous discussion of annotations, these annotations are pro-actively produced by the computer rather than being created by the user, and are designed to aid reading-while-writing rather than writing-while-reading. However, pro-active annotations can also aid reading: the other MIT work mentioned above, Margin Notes, pro-actively adds annotations to each web page the user loads (provided the user has asked for the facility). These annotations relate the new page to the user’s current work, i.e. they dynamically personalise the page.
Overall the MIT tools, and other forms of
electronic agent working on the student’s behalf, bring two lessons. Firstly
they have, we believe, huge potential to help students relate what they are
currently writing to work that has gone before. The pro-active nature of the
tools means, of course, that they achieve what paper can never achieve. Thus,
at least for reading-while-writing, there is an opportunity for the electronic
world to be better than the paper one. Pro-active enhancement is the first
of the opportunities we identify.
The second lesson is that a generalised form of annotation is a key mechanism for implementing both reading-while-writing and writing-while-reading. In this paper we will sometimes talk in terms of the general concept of integrating reading and writing, and sometimes will talk of annotation, a powerful mechanism for realising the concept.
Our second opportunity applies to writing-while-reading, and particularly to annotations (or whatever else) that are written to add value to what we are writing. The purpose of writing a annotation, whether electronically or on paper, is that the annotation will be visible when the underlying document is re-read. This re-reading may be soon after the annotation was made or months/years later. For any project that lasts more than, say, a week, and requires background reading, every student soon discovers the value of annotating the documents read. These annotations may capture the student’s opinion (‘I disagree’), relate the document to another (‘This re-enforces X’s conclusion’), or simply highlight key extracts. The annotations later prove their worth when project reports are written. The advantages of electronic, rather than paper, annotations are enhanced if all the annotations a student makes are automatically stored in a repository. Our vision at its most ambitious is that all the annotations made during a student’s education, and indeed in their later work, are captured in a repository. The vision is therefore for lifelong annotation. (We have done some preliminary work in realising this (Brown, 2003), and have looked at the nature of a repository and the disciplines that users should follow in order to get best value from their annotations: a little extra effort when preparing an annotation can add greatly to its future worth.) Material in a repository can be searched, arranged into structures, selectively retrieved, etc. A use might, for example, be ‘Find all annotations containing the word "statin" (either in the annotation itself — assuming this is textual — or in its anchor)’. The results of such a search can then be used to find the annotated documents and display the annotations in situ. The result is a facility that not only enhances the advantages of conventional annotations, but extends to encompass the functionality of bookmarks, user trails, etc. Furthermore a small practical experiment we have done indicates that the facility can be even more general. We showed (Brown & Brown, 2003b) that annotation is a form of hypertext linking, indeed each can be regarded as an aspect of a single superconcept; thus a further bird can be killed with the same stone. (Our experiment involved implementing diverse sets of annotations using only the facilities already provided by an hypertext system.) In summary, our second opportunity is to exploit the benefits of a repository of annotations. A particularly valuable instance of this opportunity is to focus on the use of annotations to help people handicapped by poor long-term memory.
This opportunity is enhanced if, at the time of creation, the student captures the nature of the annotation. One way of doing this is to allow the user to attach attributes to each annotation (Krottmaier, 2003). Another is to allow the user to create a "data type" to distinguish different annotations (e.g. "Quotation", "Implementation", "Theory", "Citation", "General"), thus indicating how the annotated material might be useful in the future, e.g. in a thesis yet to be written. Assuming all annotations are automatically stored in a repository, the student can then issue queries such as: ‘tell me all the Quotation annotations that contain the word "hypertext"’. We discuss data types again later.
A repository can also capture pro-active suggestions made by, e.g., a Remembrance Agent. Some time after the original writing, the author might want to take a more reflective look at their work, and look at suggestions made when the work was first created (these suggestions relate to the dynamic way the document was written, and thus need to be captured at the time).
Having identified two opportunities, we will balance them by looking at threats. We have identified three threats, and will look at them together. The three are:
We have implicitly assumed up to now that paper annotations apply to paper documents and electronic annotations to electronic documents. However, work at Cambridge University Computing Laboratory (Brown, Harding, et al, 1998) has allowed electronic annotation of paper documents (and, in principle, vice-versa). The work uses the Digital Desk developed at Xerox (Wellner, 1993): this is a physical desk that allows electronically-produced images to be projected on top of the desktop; it also has a camera, mounted above the desk, to capture the user’s pointing and writing activities. The projected images can be directed onto existing paper documents on the desktop, i.e. they can be annotations. The Cambridge work, with its goal of ‘Active paper for active learning’, involves providing electronic annotations on top of the paper book the student is studying. Thus the student works with a real, perfectly ordinary, book: presumably their preferred medium. One application is teaching grammar: here the electronic annotations on the book highlight certain grammatical constructions. For example the annotations might identify all the proper nouns on the current page by projecting coloured rectangles onto the relevant words. This highlighting might either be under control of the student (‘Show me all proper nouns’) or under the control of courseware the student is using. The courseware might ask the student first to try to identify the grammatical objects for themselves, and then show the correct answer; to the student the annotations produced by the courseware are pro-active ones.
The underlying mechanism is that the computer has an electronic copy of the book the student is using, and synchronises its electronic document with the student’s paper document. In the grammar-teaching application the source encoding of the electronic copy contains pre-prepared mark-up (in SGML) that gives grammatical information. This information is, in fact, structured at several levels, e.g. each sentence, each phrase, each word. This extra mark-up is used to create the annotations that are projected onto the book. In other applications, such as language translation, the annotations might be created on-the-fly — e.g. an annotation showing a possible translation of a word into another language might be generated via a dictionary look-up.
The Digital Desk can also support annotations made by the student. When creating an annotation of her current paper document, the student can first specify the anchor of the annotation: this is done by dragging over the anchor using a device that can be recognised by the Digital Desk’s camera. The student then supplies the annotation itself, either as text or by drawing. The computer remembers the annotation, and when the student next opens the same book on the same page it duly highlights the anchor, and projects the annotation onto (or beside) the page.
If we move on from writing-while-reading to reading-while writing, the Digital Desk again offers advantages. The user could, for example, write on paper and ask for a Remembrance Agent to project suggestions into the margin of the document being written.
Digital Desk technology, though not yet available as products, has huge future importance — indeed in 2003 the work won an ACM award for the paper of most lasting impact. In terms of integrating reading and writing, it lets the user do one on paper and the other electronically in any way they like — our third opportunity. This third opportunity offers a possible answer to our first threat (the user’s likely preference to read from paper and to annotate on paper). This is because the Digital Desk allows the use of paper for these activities in an almost normal way, but it can still capture and reuse annotations electronically.
When we want to share a document with others, there are plenty of easy ways of doing this. We can, for example, e-mail it or mount it on the web. There are some products to help collaborative working on documents, but they are not currently widely used — even when documents have collaborative authorship. The reason is that Computer Support for Collaborative Work (CSCW) is hard, and brings a host of new issues. Above all it is hard to make CSCW systems easy to use, compared with personal systems.
If we want to share annotations we can use the same methods as for sharing other documents. We can for example mount an annotated document on the web. The only constraint is that, both for ethical reasons and copyright reasons, it should be clear to the reader what is annotation and what is the underlying document. This constraint is normally easy to satisfy. It is desirable that another user should be able to add their own annotations to an already-annotated document, and then publish the doubly-annotated document. Again this is normally easy to do, though it begins to break down after, say, five levels of annotation, because the user interface becomes increasingly complex. This is, however, rarely a serious problem.
Although this simple sharing of annotations can work well, many annotation systems have concentrated on collaborative annotations: a group of people continually sharing their annotations. The result is like a discussion group, with discussion points attached to anchors in the underlying document. Obviously there are some educational situations where this is valuable. Moreover it has value in that readers of the underlying document can be included in the collaboration: when opening the document they can be told that annotations are available and can be shown if desired. Consequently there is a means for a reader of a document, unaware of what annotation activity has occurred, to find out what is there.
In spite of these advantages we believe that the focus on collaborative annotation is a mistake. Our first reason for this is the results of a study (see Marshall & Bernheim Brush, 2002). The study's (albeit preliminary) conclusion is that personal annotations are different from collaborative ones. The whole purpose of most personal annotations is to do them quickly, without interrupting the flow of reading, and without taking the trouble to make them comprehensible to others (and also without fear from retribution (‘This is complete rubbish’)). Thus we believe that personal annotation systems should be separate from collaborative systems — the latter are more in the realm of discussion groups. A further reason for this conclusion is that collaborative systems are necessarily hard to build and harder to use than personal ones.
To return to the overall theme of this paper, reading is most often a personal activity, not a collaborative one. Writing is likewise most often a personal activity. If we want to integrate reading and writing we should concentrate on personal reading and personal writing. This is hard enough in itself without bringing in all the extra baggage that collaborative systems bring. Our fourth opportunity, albeit a rather backhanded one, is to throw away all this collaborative baggage.
In a previous paper (Brown & Brown, 2003a) we have identified two forms of annotation. The commonly-supported form is what we call an enhancement-annotation: here the annotations are essentially icing on top of the original document; the original document remains entirely visible. The second form is an edit-annotation, which is an augmented form of an enhancement-annotation that allows deletions, insertions, and replacements in the original document. Thus a set of edits, as performed by a text editor, could be represented as a sequence of edit-annotations.
User interfaces are not a prime topic of this paper, but if edit-annotations are going to realise their full power, the user interface should, on option, allow annotations to be displayed as embedded in the original document (Zellweger et al, 2001), perhaps indistinguishable from the original (provided that intellectual property considerations allow). Thus a replacement would replace material the user sees, and a deletion would delete it. This form of interface, coupled with the use of edit-annotations, covers those applications where the user is just interested in the final product (i.e. the document generated after applying the annotations), and not in the mechanisms used to produce it (i.e. the individual annotations). In particular it covers the production of different versions of documents, where an individual version might be generated by a tutor, by the courseware the student is using, or by the student herself. Thus our fifth opportunity is to generalise the notion of annotation to include editing, and to provide flexible user interfaces that allow annotations to be displayed either in-line or out-of-line. The opportunity applies both to writing-while-reading and to reading-while-writing, though most of us would flinch if the latter leads to an agent that, while one is writing, reads other documents and then changes our own document as a result. Any addition of power to a computer system increases both the power to help and the power to annoy.
Earlier, when discussing the Digital Desk, we outlined a grammar-teaching application. One of the features of this application is that there was a large collection of potential annotations provided, covering all the grammatical constructions in the underlying document; the user (or the courseware) chooses which annotations to show. As well as offering alternative annotations at the same level (e.g. nouns and verbs), the collection of annotations covered higher-level annotations (e.g. grammar of sentences). Many other annotation systems provide similar facilities for annotation collections. It is easiest when, as with the grammar-teaching application, all the annotations form a coherent whole, prepared by a single author. If the sets of annotations are prepared independently by different authors, then there is certain to be a fundamental lack of coherence, and creating a user interface whereby the user sees at least a semi-coherent result is certain to be problematical. When the user is offered a choice, the sets of annotations need to be distinguished. This can be achieved using data types as we discussed earlier. In experiments we have performed (Brown & Brown, 2003b), we have explored the attachment of a data type to annotations, in a manner akin to data types in programming languages. It is a further aid if the data types support hierarchy and aggregation, as in object-oriented programming. Thus in our grammar-teaching example, one data type could be Part-of-speech, and this could have sub-types such as Noun, Verb, etc. Our experiments revealed some problems with these ideas of data-type (e.g. individual annotations that possess several different data types), but we believe in the merits of the idea. More generally, providing metadata about annotations can offer the benefits of data types and more.
Furthermore data types can encompass the ‘dimensions’ identified in Marshall’s extensive study of annotations (Marshall, 1998). Thus our sixth opportunity is to add richness to annotations by allowing multiple sets of annotations and by giving them data types. If we weaken this by just saying that the opportunity is to attach properties to annotations, then many existing annotation systems already capture the opportunity.
The information explosion is a huge problem in education, as in other fields. An approach to taming this is the use of agents that, knowing the user’s needs, automatically supply the user with appropriate documents — perhaps with relevant sections highlighted — and filter out unwanted ones. The problem is capturing the user’s needs: most users are reluctant to fill out forms, etc., so the needs largely have to be captured automatically. One simple aid to doing this is keeping a history of the web pages the user has read, and the links he has followed. Analysing this history — and continually re-analysing it to catch changing trends as a user moves from one project to another, for example — should reveal the user’s interests. The value of this analysis can be enhanced by looking at the user’s annotations too. When he makes an annotation the user has (normally) made an active indication that material is of interest. The value of this is augmented if annotations have data types; these can help identify the nature of the user’s interest. Indeed at its extreme this facility for indicating interest could turn the purpose of annotation on its head: a user makes an annotation not for his own immediate use, but to tell the system that certain material is of interest, and therefore he would like his agent to look out for other material of a similar nature. (In this case it is even more important that the annotation is made in a simple way, e.g. by mousing over the anchor, that does not break the user’s concentration.) Our seventh and last opportunity is automatically capturing a much more refined view of the user’s needs.
We now return to our third threat: change. We discuss this now, at the end of the paper, because it is such a ubiquitous threat. When we annotate a document the annotations are normally saved separately from the underlying document — indeed this is mandatory if the underlying document is not owned by the annotator. After the annotations have been saved, the original document may change over time, unknown to the annotator. In particular the positions where the annotations should go, the anchors, might change or disappear. (Worse, the underlying document may disappear altogether.) In this situation, there are two possible cases. In some cases the user is only interested in the original document; in others they want their annotations to be carried over to any updated version of the original. The first case is easy to handle: a copy of the original document can be preserved along with its annotations. Several projects have worked on the second case, using intelligent methods to adjust anchors when the underlying document changes. These approaches use a more intelligent method of capturing the anchor of an annotation than just saying ‘the anchor is characters 560-580 of the underlying document’ (Röscheisen et al, 1995; Bernheim Brush et al, 2001). These projects have achieved considerable success, but any approach, however clever, will break down if the underlying document is changed enough. Overall, if the underlying material is highly dynamic, integrating reading and writing is an ephemeral activity. On the other hand if the underlying material is static, as it is for example for novels and published research papers, the value of the integration can be exploited over a lifetime — and beyond, if passed on to someone else.
To take a wider view, the inability to cater for change is a threat to almost any IT project. How many web sites have fully up-to-date information and error-free external links? In our case, change is a threat not only to the opportunity to use a repository to store annotations over time but to several other opportunities too. For instance the user’s style of giving data types and other properties to annotations will evolve over time. Nevertheless the threat of change can be exaggerated: it leads to our data being imperfect, not useless. Thus just as web users accept dangling links as a way of life, users must accept the occasional wrong or lost (i.e. orphan) annotation.
Our analysis of electronic integration of reading and writing has identified three threats, all described in Section 4, and the following seven opportunities:
This paper — and our limited practical experiments to test certain aspects of the ideas — has merely scratched the surface of what we believe to be an exciting field for the future. We believe that there is a host of opportunities to produce new software tools that throw away pre-conceptions that reading and writing are disjoint activities, and produce radical new aids to students. This applies especially in higher education, where reading and writing are such important activities. As with any opportunities there are pitfalls, as exemplified by our three threats, but an awareness of the enemy is a great aid to defeating him.
Bartholomae,
D., & Petrosky, A. (1986). Facts artifacts and counterfacts: theory and
method for a reading and writing course. Upper Montclair, NJ:
Boynton-Cook.
Bernheim
Brush, A.J., Bargeron, D., Gupta, A., & Cadiz, J.J. (2001). Robust
annotation positioning in digital documents. Proc. SIGCHI ’01,
Seattle, WA. 285-292.
Brown,
H., Harding, R., Lay, S., Robinson, P., Sheppard, D., & Watts, R. (1998).
Active paper for active learning. ALT-J, 6 (1), 95-101.
Brown,
P.J. (2003). Lifelong annotation: a working paper.
http://www.dcs.ex.ac.uk/~pjbrown/annotation/lifelong.html
Brown, P.J., &
Brown, H. (2003a). Annotation: a step towards the read/write document.
http://www.dcs.ex.ac.uk/~pjbrown/papers/annotation_step.pdf
Brown, P.J., &
Brown, H. (2003b). Is annotation a form of hypertext linking?.
http://www.dcs.ex.ac.uk/~pjbrown/papers/annotation_guide_experiment.pdf
Brown,
P.J., & Brown, H. (2004). Integrating reading and writing of documents. Journal
of Digital Information, 5 (1).
Bush,
V. (1945). As we may think. Atlantic
Monthly, 176, 101-108.
iMarkup,
(2003). iMarkup: annotate, organize and collaborate on the web.
http://www.imarkup.com/products/annotate_page.asp
Kahan,
J., Koivunen, M-R., Prud’Hommeaux, E. & Swick, R.R. (2001). Annotea: an
open RDF infrastructure for shared web annotations, Proc. WWW10
International Conference, Hong Kong, 623-632.
Krottmaier,
H. (2003). Enhanced annotations. Society for Information Technology and
Teacher Education International Conference 2003(1), 991-993.
Marshall,
C. (1998). Toward an ecology of hypertext annotation. Proc. ACM Hypertext
’98, Pittsburgh, PA. 40-49.
Marshall,
C., & Bernheim Brush, A.J. (2002). From personal to shared annotation. CHI
2002 extended extracts.
Rhodes,
B.J. (2000). Margin Notes: building a contextually aware associative memory. Proceedings
of the Conference on Intelligent User Interfaces (IUI’00), New Orleans,
LA. 219-224.
Rhodes,
B.J., & Maes, P. (2000). Just-in-time information retrieval agents. IBM
Systems Journal, 39
(4), 685-704.
Ronchetti,
M. (2002). Why web page annotation tools are not killer applications? A new
approach to an old problem. World Conference on E-Learning in Corp., Govt.,
Health, & Higher Ed. 2002(1), 837-841.
Röscheisen,
M., Morgensen, C., & Winograd, T. (1995). Interactive design for shared
World-Wide Web annotations. Proc. CHI ’95, Denver, CO. volume 2,
328-329.
Wellner,
P. (1993). Interacting with paper on the DigitalDesk. Communications of the
ACM, 36 (7),
87-96.
Zellweger,
P., Bouvin, N.O., Jehoej, H., & Mackinlay, J.D. (2001). Fluid annotations
in an open world. Proc. 12th ACM Hypertext Conference, Aarhus,
Denmark, 9-18.