Standards come into being for a variety of reasons, and in a variety of ways, not always entirely explicable. They may be entirely market-defined; for example by manufacturers' attempts singly or as a group to control market share, or by consumers' desires to simplify purchase decisions. Standards also result from pressure applied by well-intentioned groups of experts, or as a consequence of legislation in the public interest. And finally, standards come about as the expression of some emergent consensus within some large community. This last method is the most likely to last, but the most difficult to achieve.

In creating such a consensus, there is an inevitable tension between the need to transform what is simply tried and tested into something normative and binding on the one hand, and the reluctance to straitjacket or constrain unanticipated development on the other. This is particularly true of the research and development arena, which depends for its survival on innovation, and thus the ability to provide answers to as yet unformulated questions, while at the same time being as concerned as any other community to codify existing practice. The research community is populated by experts, who need maximal flexibility and who distrust constraint, but also by novices, who need access to that accumulated expertise in a consistent and codified form, if only in order to rebel against it and thus become experts in their turn.

Standardization of the way in which information is stored and represented (rather than processed) is the key to a number of closely related problems, all of central concern to users of modern Information Technology, be they academic or commercial. For creators of language resources in particular, it addresses the difficulty of ensuring that information is reusable; the difficulty of ensuring that information represented in different ways can be seamlessly integrated; and the difficulty of facilitating loss-free information interchange between the widest choice of different platforms, different application systems and different languages.

By standardizing at the level of text representation, we can hope to retain the flexibility needed to develop new applications, while ensuring that old ones continue to function. By attempting a theory-neutral standardization, at the level where consensus exists, we avoid the need to reinvent the wheel, without requiring that everyone drive a particular brand of bicycle.

In this spirit, the TEI which form the topic of this paper, aim to provide not a set of normative rules for particular applications, but rather a modular and extensible framework, within which particular application-specific norms can be defined. The development of such TEI-aware norms is already underway in a number of contexts, most significantly for the present audience, within EAGLES and related EU projects such as Multext, but also in a wide variety of corpus building, scholarly editing and digital library projects. Such projects have in common the need to customize and make less generic the framework defined by the TEI, retaining as they do so the capacity for interchange for which it was developed. The general principles, and many of the specific mechanisms, underlying this approach are of clear relevance to all large scale users of information technology.

This paperAn earlier version of which was published as in OII , Nov 1994. describes the origins and organization of the TEI scheme, including some technical details of how it may be customized for multiple application areas, and an overview of its coverage.

The Text Encoding Initiative (TEI) is an international cooperative research effort, the goal of which is to define a set of generic Guidelines for the representation of textual materials in electronic form. The project was sponsored and organized by three leading professional associations in the field: the Association for Computational Linguistics (ACL), the Association for Literary and Linguistic Computing (ALLC) and the Association for Computing and the Humanities (ACH). It has been funded throughout its five years of activities on both sides of the Atlantic: primarily by the US National Endowment for the Humanities and by the European Union 3rd framework Programme for Linguistic Research and Engineering, but also with grants from the Mellon Foundation and from the Canadian Social Sciences and Humanities Research Council. Of equal significance has been the donation of time and expertise by the many members of the wider research community who have served on the TEI's Working Committees and Working Groups.

As its title suggests, the TEI is strongly interested in text. But this interest is by no means confined to the use of electronic text as a stage in the production of paper documents, and the word text should not be read too literally. The TEI is equally concerned with both textual and non-textual resources in electronic form, whether as constituents of a research database or components of non-paper publications.

Like the publishing industry, the research community has long realized that its stock in hand is not words on the page, but information, independent of any particular physical realization. As technology emerges which is genuinely adequate to the task of integrating text, graphics and audio into a seamless information-bearing vehicle, so the importance of that integrated vision becomes more apparent. By providing a description of information which is independent of realization or media, the TEI scheme, like other SGML-based approaches, enormously facilitates the construction and exploitation of multimedia technology.

In the same way, the texts with which language researchers are concerned are likely to be very heterogenous. In the construction of language corpora such as the British National Corpus See http://info.ox.ac.uk/bnc for details of this 100 million word TEI-conformant corpus of modern British English., material as divers as newspapers, books, office memoranda, playscripts, publicity brochures, letters and diaries, transcribed lectures and interviews, TV and radio broadcasts, and unscripted conversations are integrated into a single body of material. Research needs impose that this integration be carried out with minimal loss of information, and at the same time with minimal complexity: in any case, the resulting text is far removed from the conventional notion of a printed work.

Electronic texts are most obviously different from printed ones in that the former contain markup or encoding, which makes explicit various features of the text, so that they can be efficiently processed. Printed texts adopt a variety of similarly-motivated conventions (use of typeface, organization of the carrier medium etc), but these are not so readily processable as the tags of a formal markup scheme.

The goals of the TEI project initially had a dual focus: being concerned with both what textual features should be encoded (i.e. made explicit) in an electronic text, and how that encoding should be represented for loss-free, platform-independent, interchange.

Early on in the project, the Standard Generalized Markup Language (SGML; ISO 8879) was chosen as the most appropriate vehicle for the Guidelines, initially on the purely pragmatic grounds that to create a comparably expressive and versatile formal language would be a major research project in itself. In the event, despite some frequently rehearsed inelegancies, SGML has proved entirely adequate to the needs of researchers, and after five years, is still increasing its domination of the software industry, with new product announcements coming every year. The TEI was thus able to focus its efforts on the expression, using SGML, of the set of textual features indicated as its first goal above.

The prime deliverable of the TEI project is a very large number (over 400) of textual feature definitions, expressed as SGML elements and attributes, with associated documentation and examples. These elements are grouped into tag sets of various kinds, as further discussed below, and together constitute a modular scheme which can be configured to provide hardware-, software-, and application- independent support for the encoding of all kinds of text in all languages and of all times.

The TEI tag sets are necessarily based on, but not limited by, existing encoding practices; they are designed to be both comprehensive and extensible. They are collectively documented in a substantial reference manual, the , which appeared in May 1994 after five years of extensive development work. This 1400 page manual is published both in paper and electronic hypertext form and is also available over the Internet, in a variety of formats. edited by C.M.Sperberg-McQueen and Lou Burnard (Chicago and Oxford, ALLC-ACH-ACL Text Encoding Initiative, 1994). For details of current availability and locations, see the official TEI Web page at http://www-tei.uic.edu/orgs/tei.

As an SGML application, the TEI scheme necessarily requires the existence of some kind of document type definition (DTD). Current approaches to dtd design may be caricatured as falling into one of three camps, depending on their answer to the question How many DTDs does the world need?.

For many of the first users of SGML, the appropriate answer was One: the whole purpose of the exercise being to define a template against which all texts could be checked rigorously and consistently. This approach, which might be characterized by the phrase we know what's best for you, has an obvious place in applications such as technical documentation, but is equally obviously inappropriate where the object of the exercise is to describe texts produced before the blessings of structured document design were revealed to the world.

At the opposite extreme are those whose answer would be none, for whom no DTD can ever be adequate to the full complexity of the texts to be described: this attitude might be caricatured as No-one will ever understand my problem. Again, it is not impossible to imagine applications for which a DTD consisting only of elements with the content model ANY would be entirely appropriate (the first electronic edition of the provides one obvious example), although its usefulness in the general case is less clear.

Perhaps most numerous are those who shrug their shoulders and say as many as it takes: the world will always need new DTDs, in the boundary case, one per document. In the name of pragmatism, this attitude risks crowding the fledgeling possibility of information interchange out of the nest entirely; nevertheless, its popularity reminds us that sometimes the document must drive the DTD, rather than the reverse.

The approach taken by the TEI attempts to combine virtues of all three of these approaches. It defines not one, but many possible DTDs, which may be tailored to the needs of a particular application in a way difficult or impossible with most other general purpose DTDs so far developed. The user of the TEI scheme is offered the opportunity of building a DTD which matches his or her requirements, but constrained to do so in a way that facilitates interchange.

We refer to this somewhat jocularly as the Chicago Pizza model. All pizzas have some ingredients in common (cheese and tomato sauce); in Chicago, at least, they may have entirely different forms of pastry base, with which (universally) the consumer is expected to make his or her own selection of toppings. Using SGML syntax this might be summarized as follows: ]]> In the same way, the user of the TEI scheme constructs a view of the TEI DTD by combining the core tag sets (which are always present), exactly one base tag set and his or her own selection of additionaltag sets or toppings.

We use the term tag set to denote simply a collection of definitions for SGML elements and their attributes. These tag sets are the basic organizing principles of the TEI scheme, and are divided into four groups: core tag sets defining elements likely to be needed by all documents, and therefore available by default in all cases. base tag sets defining particular classes of document whose gross structure may vary; in general, only one base tag set is appropriate for a given document. additional tag sets defining sets of elements which may be found in any class of document but which are typically associated with some specialized application or detailed subject area. auxiliary tag sets comprising elements with highly specialized roles, typically for description of some part of the encoding scheme, and which make up a DTD independent of the main one. In general, elements appear in only one tag set, though the current model allows for the redefinition of elements within different base tag sets. Elements may not be defined in more than one additional tag set.

This modularization is achieved by the use of parameter entities in the TEI DTD, which is further discussed below. To illustrate the basic mechanism we present here the start of a minimal TEI-conformant document in which the base tag set for prose has been selected together with the additional tag set for linking: ]> ]]> Because this selection of tag sets is effected explicitly by declarations within the DTD subset, as shown above, any recipient of the document can tell which TEI tag sets are required to process it. Any deviations or modifications of the TEI definitions (for example, the renaming of elements, or the addition of new ones) may be made in a similar declarative manner. Once a given view of the TEI dtd has been defined in this way, it can be fixed or compiled to preclude further modification and also to remove the complexity necessarily introduced by the extensive use of indirection in the TEI dtd.

Two core tag sets are available to all TEI documents without formality. The first defines a large number of elements which may appear in almost any kind of document, whatever kind of base tag set is in use. The second defines the header, providing something analogous to an electronic title page for the electronic text.

To construct a view of the TEI DTD, the user must always choose one base tag sets. Six of these are currently defined, for documents which are predominantly one of prose, verse, drama, transcribed speech, dictionaries, or terminological databases. Another two are provided for use with texts which combine these basic tag sets.

The choice of a base tag set determines the basic structure of all the documents with which it is to be used, reflecting the fact that subelements likely to appear within a dictionary (for example) will be entirely different in kind from those likely to appear within a letter or a novel, and even more so from those likely to be found in a transcription of spoken language. To cater for this variety, the constituents of all divisions of a TEI text element are not defined explicitly, but in terms of parameter entities. The mechanism used is to provide definitions like the following within the DTD, one of which the user must over-ride by supplying an appropriate declaration in the DTD subset: ]]> The body of the main dtd contains a series of alternative definitions, each enclosed within an SGML marked section named after the base which it defines, as in this simplified example: ]&null;]> ]&null;]> ]]> Within the body of the DTD, elements are defined using these parameter entities only, for example: ]]> To select a base tag set a declaration such as the following should be supplied within the DTD subset for the document: ]]> This will over-ride the declaration within the TEI DTD itself, because it is given first. If no base is declared, the DTD will not compile.

The value of the parameter entity called component.seq will thus differ in different bases. In this way it is possible for the divisions of a text using the drama base (for example) to consist of speeches and stage directions, while those of a text using the dictionary base will consist of lexical entries.

Ten additional tag sets are defined by the current TEI proposals. These include tag sets for special application areas such as the orthographic transcription of speech, the detailed physical description of manuscript or print material, and the recording of an electronic variorum modelled on the traditional critical apparatus. A tag set is defined for the detailed documentation of contextual information needed by language corpora, as well as for the detailed encoding of names and dates; abstractions such as networks, graphs or trees; mathematical formulae and tables etc.

In addition to these application-specific additional tag sets, some more general purpose additional tag sets are defined for linking and alignment analysis and interpretation feature structure analysis

The tag set for linking and alignment extends the set of linking and pointing elements already defined in the TEI core to provide facilities for linking to arbitrary locations or spans of texts, whether or not these are in the current document, and whether or not the target is an SGML document. Mechanisms are included for recording the alignment or correspondence of parts of a text, for example in multilingual corpora, or for marking the alignment of audio or video with a transcription of it. As such, this tag set provides a usefully large subset of the facilities offered by the HyTime standard, but with a considerably simpler and more efficient interface. Witness the fact that, as of May 1995, support for the TEI extended pointer mechanism has already been implemented in Softquad's Panorama Pro, and Electronic Book Technology's DynaText --- the two market leaders amongst commercial SGML browsing software.

As noted above, a generic segmentation element is defined for the identification of textual spans appropriate to any analytic scheme. An out-of-line generic interp element may be used to link arbitrary text segments (which may be nested or discontinuous) with any user-defined set of attribute/value pair interpretations. Specific tags are also defined for the most common requirements of linguistic analysis such as identification and typing of morphemes, words, phrases, and sentences.

A specialized tagset is also provided for the encoding of abstract interpretations of a text, either in parallel with it or embedded within it. This is based on the feature structure notation employed in theoretical linguistics, but has applications beyond linguistic theory. An introduction to this tag set is provided by D. T. Langendoen and G.F. Simons ``A rationale for the TEI recommendations for feature-structure markup'' in (forthcoming, 1995; for an extended discussion of an application of the feature structure scheme to the problems of encoding historical source materials, see D. I. Greenstein, and L. Burnard ``Speaking with one voice'' (ib).

Using this mechanism, encoders can define arbitrarily complex bundles or sets of features identified in a text, according to their own methodological bias. They may thus embed a whole range of interpretations of a text, linguistic, literary, or thematic, within a text in a controlled manner. The syntax defined by the Guidelines not only formalizes the way in which such features are encoded, but also provides for a detailed specification of legal feature value/pair combinations and rules determining, for example, the implication of under-specified or defaulted features. This is known as a feature system declaration and is defined by an auxiliary tag set.

An additional tag set is also provided for the encoding of degrees of uncertainty or ambiguity in the encoding of a text. These particular tag sets exhibit in a particularly noticeable form one of the chief strengths of the TEI approach to encoding: it provides the encoder with a well-defined set of tools which can be used to make explicit his or her reading of a text. No claim to absolute authority is made by any encoder, nor ever should be; the TEI scheme merely allows encoders to come clean about what they have perceived in a text, to whatever degree of detail seems appropriate.

A user of the TEI scheme may combine as many or as few additional tag sets as suit his or her needs. The existence of tag sets for particular application areas in the current draft reflects, to some extent, accidents of history: no claim to systematic or encyclopaedic coverage is implied. Indeed, it is confidently expected that new tag sets will be added, and that their definition will form an important part of the continued work of this and successor projects.

The TEI Guidelines have taken more than five years to reach their present state, the first at which they can be said to be reasonably complete. In retrospect, it is doubtless true that they could have been created much more quickly with less involvement from the research community, or a clearer statement from it of a set of particular goals. But that statement would have inevitably limited the scope of the resulting scheme, providing exactly the kind of strait-jacket which we wished to avoid. Moreover, by prioritizing any one research agenda however well-articulated, we would have effectively disenfranchised and alienated all others. A little like the early Church fathers then, the TEI chose to provide as broad and as catholic a means of salvation as possible.

At the same time, the TEI scheme applies rigorously the principle essentia non sunt multiplicanda praeter necessitatemGenerally attributed to William of Occam (1300-1349), this recommendation is known as Occam's Razor; it may be translated as Essences should not be unnecessarily multiplied and refers properly to the distinction made by the Scholiasts between essence --- those properties of an entity which define its type and accidents --- those properties specific only to one instance of an entity. Rather than defining discrete elements for different kinds of list (bulleted, glossary, enumerated etc.). the TEI scheme defines a single list element which bears a type attribute to distinguish amongst these various kinds. In the same way, all kinds of links between document elements, whatever their semantics, are encoded using the same tags. To handle the indefinite number of elements potentially needed to handle all kinds of analysis and interpretation, a small number of generic tags are proposed which (in the case of the feature structure tag set referred to above) are sufficiently abstract and general to cater for almost any kind of interpretative judgment.

At the same time, there remain many situations in which the TEI's desire to exclude no-one has lead to a multiplication of distinctions at first sight rather bewildering. It seems to say the least unlikely that anyone will ever encode a document using every possible element defined by the union of every TEI tag set, though such a monster DTD is indeed possible.

As published, the Guidelines constitute a substantial document unsuitable for casual browsing, even in electronic form. The TEI therefore plans to make available a number of smaller introductory tutorials focused on particular application areas. Two such have already appeared: one dealing with terminological systems, Melby, Alan et al (Vienna, Infoterm, 1993) and the other on encoding of manuscript transcriptions Robinson, Peter , Oxford, Office for Humanities Communication, 1994.

A third tutorial has also recently been completed, documenting a special pedagogically-motivated subset of some 200 elements, selected from the whole TEI scheme (not just the core). Known as TEI Lite, this DTD has already been used in two electronic publishing projects and is in use at electronic text repositories at the Universitirs of Oxford, Virginia and Michigan, and elsewhere. At the time of writing, the document defining this scheme is only available in electronic form, as (Sperberg-McQueen, C.M. and Lou Burnard (Chicago and Oxford, May 1995)) from the URLs http://www-tei.uic.edu/orgs/tei or http://info.ox.ac.uk/~archive/teilite

The real proof of the effectiveness of the TEI design will come only with its wide-spread adoption, tailored to the particular needs of individual projects. As far as can be judged from the long list of early implementors, such evidence will soon be forthcoming.

This article has focussed chiefly on the complexity and generality of the TEI scheme, with a view to demonstrating its intellectual adequacy and its potential as a model for many SGML applications.

It has also attempted to demonstrate how a simple modular scheme can be implemented in such a way as to maximize the interchange space within which information interchange takes place.

The origins of the TEI scheme in the academic world mean that it has been designed with the widest possible set of applications in mind. Optimizing it for particular sets of users will be a new challenge.