Part II *

PART III *

PART IV *

CIP
Common Indexing Protocol

LDAP
Lightweight Directory Access Protocol

1. Introduction

A Development that already has taken place in the World Wide Web has started in the Directory world as well: Its success, i.e. the huge amount of Data that have accumulated, creates the need of new technologies for managing and retrieving the information. In the WWW the standard technology, i.e. hyperlink collections, were complemented by WWW search services, a meta data aware second generation of which is the current state of affairs.

 

1. Historical background

In Directory the standard means of making the data better accessible to the end user was replication. Every part of the Directory Information Tree, a logical structure that contains the data, is mastered by one Directory System Agent (DSA), a directory server. The actual information is stored in attributes. These attributes belong to entries, which are defined by the special attribute named objectClass (e.g. objectClass Person) and which represent a node in the DIT. Because of the interconnection of all DSAs, to the end user the Directory looks like one single entity. He only has to connect to any of the DSAs and can retrieve all data from all other DSAs as well, since these can interchange data via a dedicated DSA-DSA protocol called Directory System Protocol (DSP). Since this communication can happen over long distances (e.g. if a user from Europe, and thus connecting to a European DSA, wants to have Directory information from New Zealand), data retrieval can be very slow. The solution for this problem is replication or shadowing: Via a dedicated protocol, the Directory Information Shadowing Protocol (DISP) DSAs can exchange the whole data subtree of the DIT, which they master. (E.g., A DSA in UK could shadow the Data of a DSA in New Zealand). This redundant of information of course has more advantages than just accelerating user access; it also has its merits in terms of general availability in the case of failures of single DSAs. One problem that replication technology couldn't solve though was searches all over the world. Since in theory all DSAs would have to be queried only "mega DSA's", shadowing all other DSAs would accelerate such searches. But such a DSA would transcend current available software and hardware as well as counteract the distributedness of the Directory.

 

2. Previous work

In the White Pages IETF Meeting in Nov 1993, as documented in [RFC1588], the need for Directory indexing technologies was seen for the first time. Paul Barker took up this challenge for X.500, with a concept of replicating only selected attributes (e.g. email address) of selected entries (selected by the object class) to build so called Index DSAs [Barker]. Although this reduces the problems of the before mentioned "mega DSA" this approach still would not scale. Another approach had been developed in the whois++ world [RFC1913], being to reduce the information, by tokenisation into the so called centroids, which were managed in a hierarchical structured mesh of whois++ index server [RFC1914]. David Chadwick took up this idea and defined a similar approach for the X.500 world, defining Centroid-DSAs holding information about normal DSAs [Chadwick]. Both update approaches based on the classical centroid model have as drawback that the information about interconnection between several attribute values get lost through the tokenisation. The final Approach for Directory indexing was made in the frame of the IETF WG Find, and is called Common Indexing Protocol (CIP) which defines an overall index server model [CIP arch], MIME objects for index object exchange [CIP mime], a transport protocols [CIP trans] as well as an index object type dedicated for directory information [CIP TIO].

 

3. DESIRE I

Since the indexing work in DESIRE I project was exclusively web oriented, the problem of Directory indexing was not addressed. Nevertheless some of the work in DESIRE I [Lundberg] has impact on the current approach in two ways. First the general concept of DESIRE I - crawl data, index data, retrieve data - is applicable to directory indexing as well. The second impact of DESIRE I (and of the ongoing web indexing work performed in DESIRE II) is the idea to define a general index tool box that is capable of dealing with web information as well as with Directory data. Another area of common interest is the definition of metadata. On the side of the Directory part it takes place on the level of object class and attribute definitions.

4. Summary of CIP

CIP is the attempt of the IETF find WG to define a common indexing mechanism for a multitude of Internet data protocols. It defines the process of passing indexing information from server to server to make them capable of routing client queries to the servers that have the apropriate data. The only prerequisite for a data access protocol to benefit from CIP is to have some kind of referral mechanism, like, e.g. X.500, LDAP and Whois++.

 

1. Index objects

To facilitate application domain specific indices, CIP index objects are abstract and have to be defined according to the specific usage. The different requirements for index objects are handled by encapsulating the index objects within MIME wrappers [MIME] as a standardized way to specify the differences. The respective MIME type tree is application/index. The basic protocols for moving index objects are common to all objects. In general an index object contains a subset of data in a compressed form (through tokenization).

2. Tagged Index object (TIO)

A separate CIP document [CIP-TIO] defines the TIO, which is an enhancement of the Whois++ Centroid concept. The added tags are attribute identifiers that allow to reconstruct the whole entry from the attributes contained in the index object, i.e. to "tie the individual object attributes back to an object as a whole". The data necessary for an exchange of index objects are defined as well. A TIO consists of three parts: a header, the schema specification and the actual index payload. The schema part includes the definition of the type of tokenization.

3. Data set identifier (DSI) and Base-URI

The DSI uniquely identifies a given dataset and is chosen from any part of the ISO/ITU OID space and thus consists of an unbounded sequence of unbounded integers.

Wrapped CIP index objects include a base-URI, which is needed for referral generation based on searches in the index object. The base-URI will include information about the server from which the index object was created. Base-URI and DSI are both not part of the TIO, but are given as parameters to the MIME types and thus exist in the MIME transport wrapper.

4. CIP transport protocols

As CIP relies on interchange of standard MIME messages for all requests and replies, the CIP messages are passed over a bidirectional transport system. CIP defines three modes for the transport of the index objects: 1. The stream transport, where CIP messages are transmitted over bi-directional TCP connections via a simple text protocol; 2. Internet mail infrastructure as transport, using the Reply-To header to establish the return path; 3. HTTP transport, where transaction is performed by using the POST method to send requests and responses.

5. Security options

CIP defines several possibilities to secure the transport of tagged index objects: PGP/MIME [RFC2015], S/MIME [S/MIME], SSLv3 [RFC2246]. They should be chosen in accordance with the transport mechanism. The first two security mechanisms effect the application level and thus can be used independent of the transport protocol.

 

6. Problems of CIP

1. Deployment of CIP

1. The main concept

The main aim of this work package is to define a directory indexing system that is deployable in a European context, making directory information available to the research community of all participating countries. Although aimed at Directory indexing the model should be applicable to other indexing problems.

To implement an indexing system on such a large scale, a hierarchical index creation and distribution is necessary for overall performance and scalability issues. In such a model index servers located at higher levels of the hierarchy gather the index objects of server located on lower levels of the hierarchy. For example the index server of an organisation collects the index objects of all departmental directory servers, the index server of a country collects all index objects of the organisational index server. This ends up in one root index server that includes the index objects of all country level index servers that are part of the indexing system. Since it is not advisable to have one single point of information retrieval to which all clients that want to retrieve index information would have to connect to, the collection of index objects has to be redistributed downwards the same hierarchy. Since the management of such a big collection of index objects requires a considerable amount of hardware power they will not be distributed down to the single server, but might only reach to country level.

The whole indexing system of gathering and distributing and searching index objects should be managed by the server side. Clients should not need to have special features for retrieving the index information, which means that an index server has to respond to a client the same way a normal server would do, in case it doesn't have the requested data: It just gives back a referral to a server that might have them. In the case of an index server the probability that the referral points to a server that has the data is very high. That is the only difference for the client. Although every client capable of chasing referrals could be used in the proposed indexing system, a client that includes special index related features is favourable for this project out of several reasons. First of all by now there is no publicly available LDAPv3 client that could be used for a proof of concept. Secondly special problems of index query, like the possibility of a huge amount of referrals have to be dealt with. And last but not least an index aware client can provide a better user interface that gives index specific information. Out of these reasons a dedicated client is in development, which is described in two further documents, one on specific requirements [Client-Req] and one on the client architecture, taking these into account [Client-Arch].

The mechanisms proposed in this document can be described as a subset of the Common Index Protocol (CIP), which is seen as the future standard for indexing in the Internet. Though not all features defined in CIP are planed to be implemented the overall structure should be compatible with this standard.

 

1. Gathering of Index objects

The atomic entities of the indexing system in its first stage are the index objects of the single server that are included in the indexing system. These index objects will not be modified in their content while their transport up and down the hierarchy; they will not be aggregated to bigger index objects. Although such an aggregation is defined in CIP, it produces in combination with the TIO hardly manageable problems. Through aggregation the tags of the TIO would change, which makes the retrieval more difficult. Since the index object includes header information about the data server (DSI, base URI), retrieval would have to follow back the steps of aggregation to finally reach the LDAP server. The update of index objects again would be difficult in terms of retrieving the right index entries in the right index objects, where again the whole aggregation path has to be followed. If, as proposed here, the index objects are not changed, the case of an update is quite straightforward: a new index object is to be produced and the old index object just has to be replaced in the index object collections. The DSI provides a perfect means for the identification of the index object to be replaced. Incremental update of single index objects is included in theTIO definition, which allows you to specify data blocks for add, delete and update operations. To unambigously identify the record for the delete and update oprations a unique identifier of the entry must be included in the index object. In the case of LDAP directories this identifier would be the whole untokenized DN. In a first approach the DESIRE II index system will not use this feature of incremental updates.

The index objects can be built by dedicated crawlers that crawl through the DIT subtree of one server to collect the data. A TIO converter can then in a second step produce the index object from those data. The decision which of the entries to crawl and which attribute values to collect, has to be done by each participating organisation, the maintainer of the single server respectively. These definitions should be made via crawler access policies stored in the directory itself and understood by the crawler. A separate document will define the mechanisms and the storage model for such a crawler access policy, similar to the robots.txt and robot meta tags defined for the WWW [Crawler]. To make sure that only crawlers compliant to this policy mechanism are able to get the data, the crawler has to authenticate itself. In a first stage, the crawler could be directed via access control mechanisms inherent in the Directory (acls). With such a mechanism in place it becomes irrelevant in terms of privacy issues, who will maintain and run such a crawler. It could either be the organisation itself, the NRN for all or a subset of organisations in a country, or even the maintainer of the central index objects at the root of the system.

The single servers that are part of the index system will be registered. Registered server will be put in a list, which will be accessed by the crawler or the maintainer of the crawler to retrieve knowledge about host and port of the server. Automatic server location as defined by the Service Location Protocol [SLP] will not be used, although such a technology could well be a future replacement of the registration process. The details of the registration process will be defined in a separate document [Registration].

 

The format of the index should be the TIO. The advantage of the TIO is, that all the indexed attributes of one directory entry can be identified, and search filters including more than one attribute can be used. The Dataset Identifier (DSI) should be used to uniquely identify a given dataset among all datasets indexed. All index objects should contain a base-URI in its header, which is crucial for generating referrals to the complete data of an indexed entry.

 

2. Distribution

To prevent a single index entry point, where all the worlds’ clients would connect to, the gathered index objects (TIO collections) have to be distributed downwards again. Every country level should provide index servers for the respective country index, as well as for the mega index. If appropriate, the country level indices could be distributed to several index servers at different locations in the respective country.

The downward distribution of the indices, as well as the upward sending of the indices to be aggregated can be performed via simple FTP transfer for a proof of concept. More advanced transport mechanisms defined in the CIP Transport Protocols draft can be used instead eventually.

 

3. Query routing

The clients should not have to provide special features for using the index system. It connects to an index server in the same way it would connect to any other directory server. The access protocol is plain LDAP (v3). The server should then perform the following algorithm:

Perform a search in the locally stored dataset, and return the data if found. If no data matched the search filter, the server should consult the index server to search for appropriate entries and return the referrals to the entries, based on the base-URI found in the index.

The user could influence this algorithm by adding a base DN which defines the entry point and limits the search. The user can herewith, e.g. start the search from the root level, or from any other level in the hierarchy. In any case the client does not have to know anything about the indexing system except the hostname and port number of one nearby server, which is a part of the index system.

For more information about the client side see [client-req] and [client-arch].

 

4. The over all concept

2. Security considerations

 

1. Personal data and privacy legislation

Since white pages directories contain personal data (i.e. e.g. name, email address, telephone number), it is important to conform to European privacy legislation. Even if all the data are public data and published in the directory with the consent of the affected persons, it is against that legislation to make available a bulk of such data. While transfered from one server to the other the index objects are vulnarable to get stolen by commercial data brokers and spammers. It is therefore neccessary to protect the index object data while transfering them on the net.

 

2. Encryption of the index objects

1. Authentication between servers

1. References