[[page created automatically from word-processed document; for original see: Postscript version]]
P.J. Brown, Heather Brown
Computing Laboratory
The University
Canterbury
Kent, CT2 7NF
ABSTRACT
Most hypertext systems used in the field embed some form of mark-up in each hyperdocument in order to represent the hypertext structure. Indeed, more generally, most document preparation systems use this approach. Hypertext researchers, on the other hand, say that the structure of a hyperdocument should be separate from its content. This paper investigates whether the two approaches, embedded v. separate, are really at odds with one another, and describes a technology for combining some of the benefits of both. .OK Hypertext Mark-up Guide WWW CD-ROM UNIX Microcosm Hyper-G
Mark-up of any document can either be embedded within the document or stored separately. In the hypertext field, all the expert researchers will say that separate mark-up is the only respectable approach -- the structure should be separate from the content, whereas virtually all the hypertext systems that are widely used in the field are based on embedded mark-up. This paper examines whether the issue of embedded versus separate mark-up really is an important one.
We will start with four background points -- highly diverse in nature -- which will throw light on the central issue.
In any field the `experts' despise the choice of the masses. At the time of writing the best-selling book in the UK is a thriller by Dick Francis. If, however, you suggested to Professor of English Literature that he should replace his course on, say, the works of T.S. Eliot by a course on the works of Dick Francis, you are unlikely to get a reply that treats you as an intelligent being.
The dichotomy between the experts and the masses applies in computing as much as any other field. Try suggesting to a University Computing Science department that BASIC be made the main programming language that is taught.
Thus the clash between the hypertext experts and the masses should not come as a surprise. Note that in each field the expert's choice requires more effort to understand: sometimes, as in the T.S. Eliot case, it may be beyond the comprehension of many.
A lot of us feel that small-is-beautiful is a good guiding principle for computer software. Indeed one of the reasons for the original success of UNIX was it encompassed this principle. The UNIX philosophy was (is?) that monolithic software tools are wrong. Instead you should have a collection of small tools, each good at one particular task, that are easy to connect together so that a set of tools can co-operate in performing a task. In particular UNIX provides the pipe as a connecting mechanism, though pipes work at a low level (an unstructured stream of bytes).
If this philosophy is applied to hypertext systems then hypertext systems should just do hypertext. They should not, for example, offer clever editing facilities (pattern matching and so on) since a specialist editor would do the job better. As a corollary, it should be easy to switch between the hypertext system and the editor and in particular the editor should be able to process the hypertext system's data formats.
An extension of this point is the `ubiquitous hypertext' feature we mention later.
A recently posted troff document reads: `submit your contribution in the following form:
Your title
...'
Devotees of troff will recognize a familiar trap here: the content contains lines such as .NH and .LP that are troff mark-up lines. The author therefore needs to mark these lines in a special way so that they are treated as content, not as mark-up.
The same point applies to any other embedded mark-up: unless the character set used in the mark-up is separate from that used in the content, there is a danger of parts of the content being mistaken for mark-up.
Donald Knuth [6] has championed an approach to program documentation called `literate programming'. In this approach the documentation of the program, with its embedded mark-up, is itself embedded in the program, the idea being that you can use this composite whole to produce a well-formatted and, one hopes, literate explanation of the program. There are tools to extract a formatted document from the composite, and alternative tools to extract the program itself in a form that a compiler would accept.
Knuth's idea can be generalised to cover other pairs of related texts, e.g. specification and design. The essence is that by interleaving two (or more) related documents, you make it easier to see the relationships between them.
On the face of it, Knuth is compounding the sin of embedding.
These background points will come into play later. It is now time to start the basic exposition of this paper by explaining some terminology, in particular mark-up and content. The mark-up delimits and describes the logical objects of the content. The content is the underlying text, pictures, video, etc. Actually, the mark-up is sometimes concerned with physical appearance, e.g. `switch to bold face', but we shall concentrate on the logical level here. Mark-up can be at several different levels:
It is worth saying a bit more about what we have called calculated mark-up. If the content is known to be in a form where logical objects within it can be identified, the way is open for the calculated mark-up to recognise and use these logical objects.
Examples where there is potential for the new mark-up to use knowledge about the structure of the content are:
The general point that these examples bring out is that if mark-up is specially designed for one sort of content, PostScript, say, then it can take advantage of its knowledge of the logical nature of that content, and refer to the content in terms of its logical objects using some form of calculated mark-up.
Finally, as an added complication, the mark-up can itself contain extra material that is to be added to the content. Thus the mark-up may specify the name of a button that is to be added to the underlying content.
The case in favour of separate mark-up is certainly not just an academic one. Some solid advantages are:
Interestingly many of these advantages apply to mark-up in general, not just to hypertext mark-up. For example, if one had some raw text on a CD-ROM it would be useful to be able to add separate mark-up that specified, e.g., titles, paragraphs, citations, etc., within it. This mark-up could then be used by a formatter.
The compensating disadvantages are just as solid:
The paper so far has presented a dichotomy between two opposing approaches. It is now time to change tack and consider whether, indeed, the approaches are as opposite as they seem. Can you combine the advantages of each?
The first point to make is that, to many users, the difference of approach is an irrelevant detail. How many users (as distinct from authors) of WWW, for example, know what its mark-up is like or whether it is stored separately from the content? In fact it is not. Even authors do not need to be aware of the difference: if authors do their work through a WYSIWYG interface they too are unaware of what the mark-up is like and whether it is embedded. The main constituency that remains are those people who (a) want to add mark-up to existing material that does not belong to them, or (b) want to use the marked-up material with a tool other than the one that created the material. These people need to know whether mark-up is embedded. They are currently a minority, though you could argue that if good facilities were provided for such people their numbers might grow.
We now move on to the question of whether, for those users to whom it does matter, the advantages of separate and embedded mark-up can be combined. This can be done by giving such users a choice, when they save a document they have created, of whether to make the mark-up separate. Moreover the user should be able to change their mind and convert from one form to the other, as the need arises. To be realistic the choice must be confined to a choice between embedded mark-up and separate positional mark-up. We must forego the advantages of calculated mark-up, because of the difficulty of generating such mark-up automatically from a lower-level form. To achieve our aim we propose two simple software components, a Joiner and a Splitter. The Joiner takes two input files and combines them into one, outputting the result. The Splitter is the complement of the Joiner, splitting an input file into two separate output files. Clearly the Splitter needs a policy to decide how the split is to be made. A policy of relevance here, when applied to an input file with embedded mark-up, is to output the mark-up in one file and the rest (i.e. the content) in the other, thus producing separate positional mark-up. Each different form of mark-up will need its own policy. The Joiner will likewise need rules for joining its two input files. A simple rule is to regard one input file as pure content and the other as a set of edits to the content. Thus the second file might be a set of edits to insert mark-up into the content. The policy of the Splitter is designed to fit with its complementary Joiner. Thus for the sample Joiner quoted above, the policy of the Splitter would be to extract the mark-up and output it as a set of edits to the content file. Indeed in this case the Joiner might be a simple old-fashioned text editor with commands of the form `Insert ... at position ... '.
The Joiner and Splitter are conceptually general tools, but, since each form of mark-up will have its own policy, it will have its own version of a Joiner and Splitter. Thus a Splitter designed for SGML mark-up would not work for troff mark-up. Obviously the world would be a better place if all tools shared a common style of mark-up, but this is not so now, and probably never will be.
For each style of mark-up the Joiner and Splitter should match in the sense that if you split a file and joined it again you should end up with something identical, at least semantically, with the original.
The Joiner and Splitter should basically be simple, but there is a potential complication illustrated by our Background Point 3: problems associated with content being mistaken for mark-up. Assume that we have some material stored on a CD-ROM and want to add some mark-up to it. We use a Joiner so that the resultant output can be processed by a tool that understands embedded mark-up. What happens if the material on the CD-ROM already contains lines that look like mark-up? Such problems, though introducing tedious extra complexity are, however, fortunately solvable in practice. Ideally the Joiner and Splitter should be simple and fast enough to work on-the-fly, without any need for caching.
Our scheme above involves splitting into two separate files. A further element of complexity is introduced if we want to split into more than two separate files: this would be needed if there were several independent sets of mark-up on the same content, and/or if a single mark-up file brought together several content files. Rather than exploring this complexity, however, we propose a system of layers, whereby the output from a Joiner at a top layer can be used as one of the inputs to a Joiner at a lower level, and so on through as many layers as necessary. At the complementary splitting stage the Splitter would likewise feed its output to another Splitter, with a policy as to what is split off at each layer. Figure 1 shows how the joining might be organised.
Link to picture
Figure 1: two layers of mark-up
We will now proceed to give some practical details.
A Joiner and Splitter have been implemented within the UNIX version of the Guide hypertext system [2]. Guide uses embedded mark-up and the purpose of its Joiner and Splitter is to remove some of the disadvantages of this. From its inception, Guide has had facilities for generated mark-up: for example UNIX manual pages and other troff documents can be automatically converted into Guide form on the fly, with titles converted into buttons, `SEE ALSO' items into links, etc. Hence the concept of having a program send its output into Guide is well provided for.
Guide users create hypertext facilities in a WYSIWYG way, and the authors (and users) are unaware of the underlying mark-up; the use of the Joiner and Splitter was therefore designed to be behind the scenes so that the authors and users can retain their blissful ignorance. All the author needs to do is to select an option, when it is needed, to save mark-up separately. Behind the scenes this involves Guide's Splitter, which has the policy of producing a separate (positional) mark-up file, which we call the joiner-edit-file. The joiner-edit-file has a special comment line on the front, so that, if it is fed to Guide, Guide knows that it should call up its Joiner. The initial comment line of the joiner-edit-file also identifies the content file to be used (in fact there can be multiple content files, but we will suppress such detail here). The rest of the lines have the general form:
Read N1 bytes from the content file Insert the following: ... Read N2 bytes from the content file Insert the following: ...
(Thus the mark-up is, unfortunately, in terms of low-level byte offsets -- something of a necessity, given that the content can be anything at all. If Guide were geared towards a special form of content, which had known structure, it might be possible to go a little way towards calculated mark-up.) The content can involve media other than text, but currently mark-up can only be inserted within the textual parts.
The Joiner is a crude editor that processes edits of the above form, and as a result converts back to the original form, with its embedded mark-up, and outputs this into Guide.
In fact the joiner-edit-file also contains a designator giving the policy by which it was created (we will soon see that several possible policies may be available). When a user comes to save a file, Guide by default saves it as embedded mark-up if the original input file used embedded mark-up and with separate (positional) mark-up otherwise. In the latter case the same policy is, by default, used as was found in the original input. Again, the ordinary user does not need to know whether she is using embedded or separate mark-up, but the user who cares can change the policy that is to be used on saving.
The joining and splitting described above apply both to files that are loaded when Guide starts up and to files loaded as a result of following links.
There are many cases when the user wants to go beyond the simple policy of just splitting off the mark-up. In our CD-ROM example, the user may wish to add content to the CD-ROM: extra text, button-names, icons, etc. Guide has a facility for the user to define logical objects in the source file. These might be objects such as Title, Part-number, Program-listing (when a piece of program is embedded in a document), etc. A separate table specifies for each logical object, how it is to be displayed (e.g. using X font and Y colours). These logical objects are, in fact, called contexts.
When saving a file, Guide uses the system of layers mentioned earlier. The default policy on saving is to have everything in the same layer (i.e. content with embedded mark-up), but other possible policies are to have a content layer and a mark-up layer above it, or to go a stage further and to have many separate layers. It is the contexts that provide the flexibility in assigning objects to layers, on top of a basic facility to specify whether mark-up is, by default, in a separate layer or not.
To provide this flexibility, each context has an attribute (changeable at any time by the user) which specifies what layer it and its content belong to. For example the user could create a context called Annotation which had the attribute that everything within an Annotation was saved at a mark-up layer, i.e. an Annotation is an object at the mark-up layer. Given this, the author can load one or more CD-ROM files, add some Annotations (and perhaps some ordinary Guide buttons too) and save the result so that everything lying within an Annotation context plus the Guide mark-up go into a separate joiner-edit-file that can be applied to the CD-ROM content.
As a refinement, another context could be defined, called, say, MetaAnnotation, which allowed further annotations to be added and saved at a layer above the layer with the original annotations. We then have three layers:
Any of the above three can be used independently: i.e. the user can ask for just the content, the content with Annotations, etc., or the content with both Annotations and MetaAnnotations.
Guide's Joiner and Splitter can be used in UNIX pipes when Guide files need to be communicated to other UNIX tools. As a refinement it is possible, in these cases, to adjust the Splitter's policy so that different logical objects can be selected. For example assume the user wants to extract the content layer, but minus any occurrences of a Comment context that occurs at this layer, but instead including all occurrences of the Annotation context, which normally belong at a higher layer. This can be done by temporarily adjusting the layer to which the Comment and Annotation contexts belong, and then doing a save, which passes the output to the next component in a pipe. (Indeed Comments could be assigned to a null layer in this case.) This goes towards the kind of flexibility we mentioned when discussing literate programming.
The lessons from the Guide implementation are:
We will now return to the general case and look at how marked-up files can be used by different systems, and how the Joiner and Splitter can help. Our discussion is based on a UNIX environment where tools (ideally simple uni-purpose ones) are connected together using pipes. The discussion applies equally well, however, to any operating system environment where tools can be connected together.
The simplest tool, as regards its input/output, is one where input goes in one end and output comes out the other. Most UNIX tools are designed like this, so that they can be fitted into pipes and more generally can use all UNIX's input/output redirection mechanisms. With such tools it is a trivial matter to incorporate Joiners and Splitters: the pipe mechanism is made for it.
With more complex tools the input file can link to other input files. We call this an embedded-file-link. Embedded-file-links occur extremely frequently in hypertext systems -- indeed some people argue that they are the essence of hypertext -- but they can also occur with many formatters (a document that contains a request to include another), programs (e.g. #include directives in C), etc.
Since embedded-file-links are at the core of many hypertext systems we will discuss them in more detail. Let us assume that the user wishes to add mark-up to an existing file F, which is referenced by embedded-file-links. We will also assume that the added mark-up is for a hypertext system and that this hypertext system is based on embedded mark-up -- this is likely to be the most common situation in practice.
Two properties of F are relevant:
As examples of (b):
(There may be cases where some embedded-file-links to F are to be interpreted as (b1) in some environments and as (b2) in others, but we will continue our policy of ignoring complications such as this in this paper.)
Case (b1) is easy. If, for example, a C includes a file called mylib.h, the author can create a separate mark-up file called, say, mylib.mark. When the user wants to view the file in its marked-up form he calls it mylib.mark. Ideally, if the hypertext system works on embedded mark-up, it will have an automatic mode recognition capability that automatically calls its Joiner behind the scenes to combine the two files mylib.mark and mylib.h. It is unlikely that there would be any existing hypertext links to mylib.h, but if there were they would lead to the original, unmarked-up, mylib.h.
Case (b2) is easy only when F is writable. When it is writable, the original file F can be moved to a new file FC, which will be treated as a content file, and a mark-up file, which applies to FC, can be put in place of F. Thus F is now a mark-up file, which leads indirectly to FC, the original content. If the hypertext system that deals with the mark-up has automatic mode recognition, then, on seeing any reference to F it will treat is as a mark-up file for FC.
The difficult case is case (b2) when F is not writable, because the trick of replacing F is not applicable. What is needed, either within the hypertext system or within the operating system it runs under, is a general aliasing system. In its simplest form this takes the form `use X in place of Y', but an ideal, which is simple and more flexible, is to be able to say to the hypertext system `whenever you load a file X, call the program P (which can be specified by the user) with X as argument'. In the latter case the default version of P will simply use the file X unchanged, but P could be a Joiner with an associated table with entries of form `if the file is F, apply the Joiner to it using mark-up file FM, and output the result'. Moreover this approach opens the way to a generic form of annotation: program P could for example add an extra hypertext link on the front of all pages it loaded.
If none of the forms of aliasing that we have mentioned is available in practice, system hackers may be able to provide the equivalent by installing personal modifications to the I/O libraries that open files, particularly if such libraries are dynamically linked.
Finally, let us balance our emphasis on hypertext systems that have embedded mark-up with an example that assumes a hypertext system with separate mark-up.
Let us assume, therefore, that a hyperdocument is represented by one content file with separate mark-up. We wish it to do some complicated edits on this, and decide to use a specialised editor, not the hypertext system, to do the job. Assuming we have a Joiner and Splitter that cater for the mark-up used in the hyperdocument, we apply the Joiner and send the resultant output, which is the hyperdocument with embedded mark-up, into the editor. We then do the necessary editing, and when we come to save the result, we send the editor's output to the Splitter, thus getting back to the separate form we had before.
A few points about this exercise:
.ne 6
The paper has set out to prove that the issue of separate versus embedded mark-up is not a central issue in hypertext. Most hypertext systems today are based on embedded mark-up, but they can be made to work with separate positional mark-up by using the Joiner and Splitter technology. Likewise systems based on separate positional mark-up can be made to support an embedded form if this is required, e.g. for exporting to some other tool that would like to see an embedded mark-up.
The big issue is not the nature of the mark-up but the facilities for integrating software components. In any real-world situation the hypertext system needs to work with other systems to provide a solution to a problem. The closer these systems fit together the better the solution. Our Joiner/Splitter technology requires a relatively small degree of integration. The real challenge is to proceed to a degree of integration where we never need to distinguish between documents that are hyperdocuments and those that are not, but where components work together to make hypertext ubiquitous, with any underlying tools to change the formats of the information being invisible to users.
I am grateful to the anonymous (?) referees, especially to Wendy, for helpful comments.