CHAPTER 12 - Using WAIS with CGI

How hard should it be to find information? Your time is valuable, and having to look through data that doesn't interest you uses up time that could be better spent elsewhere. Consider the number of people setting up their own Web sites every day, whether on a corporate or individual level. You get an idea of just how much data is being added on a daily basis. It's impossible to look through all that, even if you have a large pool of resources, because of the sheer volume and the frequency of change to material you've seen previously.

Now consider people in a similar situation coming to your site. They want information, and they want it now-relevant information on demand. If you meet their needs, you've improved perception of your services. If you can accomplish that without too much work, that's all the better.

By using Wide Area Information Systems (abbreviated WAIS and pronounced "ways"), you can meet these information needs with minimum effort, regardless of the platform you're running on or the kind of information you're making available. All your data can be quickly and easily indexed, and any user can then get search access to the data through his or her preferred browser. Everything is done in plain language-no fancy terms or odd parameters-and information that matches your user's needs can be presented to your user in a variety of methods.

In this chapter, you'll become familiar with

• Creating your own WAIS database
• Learning how WAIS works
• Understanding WAIS gateways: server-integration versus scripts
• Obtaining and installing WAIS software

The first step, then, is to take a look behind the scenes to understand who created the WAIS standard and why.

There's a lot of information out there. Every day something new is added to the accumulated pool, whether its our own store of knowledge or some database growing to infinity. It's a lot to keep track of, but we sure try. Out of all that information, though, only some may be of interest to you at any given point and time. You may not care if it's snowing in New York, or how far a catapult could toss a head of lettuce, but someone out there does. Some day that someone might be you. And then wouldn't it be nice if there was an easy way to sort through it all? Some companies thought so.

WAIS Origins

In October 1989, a group of companies composed of Dow Jones, Thinking Machines Corporation, Apple Computer, and KPMG Peat Marwick saw the need for an easy way to provide text-based information systems on the corporate level and decided to do something about it. Their goal was to create an easy-to-use, flexible system for searching large amounts of distributed information in various formats built on an established standard.

For ease of use, they decided that instead of cryptic commands and proprietary interfaces, the users should be presented with a consistent access method on every platform. Because searching for information normally revolves around a keyword or concept, the easiest derivative of the access method would be a block where users could type in a word or phrase. Building on that, other interfaces could be constructed that gave lists of choices for the keywords, as well as choices for what particular database the user wanted to search.

The ability to select what to search is a definite advantage. One of the anticipated uses was for electronic publishing in wide distribution; therefore, the number of data sources someone might want to search was unlimited. After all, the goal of the system wasn't just for external people to be able to find information you make available, but for you to gain access to other systems' data as well through the same procedures. In this way you could transmit your query to a remote server if you couldn't find what you were looking for where you were. So if you were to query server A, looking for CGI libraries, it might return a reference to server B. You could then immediately repeat your search on server B, and so on.

If you were interested in data on only a single computer, selectable data sources come in handy as well. How many sites have you been to that allow you to do an overall search of the site's information, as well as narrow down the field to something like product updates? Quite a few are out there, because it's natural to want to process the information-you may be looking for something that's in a definable category, which reduces the amount of information you need to sift through.

For example, look at how information can be connected when dealing with a WAIS server: multiple clients all going to one WAIS server, and that WAIS server in turn going to multiple data sources. Those data sources, in turn, could be (or could be connected to) other WAIS servers. See figure 12.1 for an example.

The benefits of WAIS are ease of use (for clients and developers), full-text search capability, and support for a variety of document types. It also has a far-reaching knowledge base; it can draw on remote databases to continue the query by example started in one location. Using results from one search can lead to a more appropriate server, and so on until the desired result is found. The drawbacks can be grouped into WAIS's incapability to support relational functions, other than relevance feedback for similar documents.

When asking yourself whether WAIS is a good fit to your situation, consider the following questions:

• Is this data already indexed in some other form (spreadsheet, database, and so on)? If you can use a database front end that might make more sense for your situation, you may want to consider it. Examples would be for order or inventory queries.
• How much data is involved, and how often will it change? If you have a large volume of text that doesn't change frequently, it would be a good candidate for WAIS. If it does change frequently, you may want to look into automatically reindexing the files on a regular basis.
• What type(s) of data are primarily referenced? WAIS is strongest in dealing with large amounts of text, as that was the goal in its design.
• Is full-text search the desired search method? If the documents and other information are already being tracked through some other database, such as a document inventory on a relational database system, WAIS implementation can add value by providing a full-text search of documents where that's desired, without adding the text of the documents to the relational database. If searching of that type isn't needed, an implementation that gives direct access to the other indexing method may prove more beneficial.
• Will multiple servers be involved? WAIS communicates easily with other WAIS servers and other locations. For single machines or limited access it may be overkill, but it excels in wide-area distributions.

If a large volume of text data is being tracked and it changes all the time, WAIS may be one of the best solutions. Indexing is quick and painless, and it encompasses the entire document rather than just keywords that need to be updated or accurately maintained. Frequent modification of a small segment of data doesn't preclude the use of WAIS by any means, but it opens up other database methods that might provide features better suited to the situation.

The type of data being referenced is very often text; however, in the case of graphics files, there's little benefit in a WAIS search other than finding file names. Querying by even more advanced methods, as some larger companies are moving to do in their search technology, will eventually provide more value by letting users visually or audibly specify patterns or colors that they're looking for: You might click a plaid shirt pattern to find shirts in a manufacturer's database with similar patterns or colors; or you might select a region of a picture to find other images with similar components (such as a bridge, water, or rivers). Although this type of data could be replicated in a descriptive file that served as a companion for each image, the manual creation of that other file would belie the purpose of being able to index the graphics by themselves.

If you were creating a customer database to track who ordered what, how much it would cost, and when it would arrive, WAIS wouldn't be the ideal candidate. On the other hand, if you had a sheaf full of technical documents for customers and were constantly adding things and revising old ones, WAIS would be perfect. Each situation brings with it a little addition or angle on a specific need that not every system will be able to address. By knowing whether your own needs are compatible with what WAIS provides, a combination of WAIS and something else, or just something else entirely, the chances that you'll get the right search system with the least amount of work on your part are greatly improved.

Think about a library's card catalog. Rather than duplicate all the data from every book, the card catalog mentions key references to help you conduct an organized and efficient search. The advantage that a WAIS database has over an old-fashioned card catalog is that even though the card catalog can contain only summary information about the documents it knows about, WAIS provides a search method that includes the contents of the documents as well as their summary information.

WAISINDEX: The Database Maker

Because a WAIS database is really just an index of documents, creating the database is a matter of going through each document and creating tables of words from the documents, titles, document locations, and other data that the search program can reference later on. The capability that WAIS has to be flexible in what people can search through leads to a number of different file types supported for indexing (or parsing). The utility that does this indexing is, appropriately enough, WAISINDEX. Some of the more common formats it can parse are included in table 12.1. Table 12.1 Common Parsing Formats Supported by WAISINDEX

File Type	Description
bibtex	bibtex/latex format
dash	A long line of dashes separates document entries within a file
dvi	DVI format (Device Independent Printer output)
gif	CompuServe Graphics Interchange Format graphics (file names only)
html	Hypertext Markup Language
para	Each paragraph separated by a blank line is a new document
pict	PICT graphics (file names only)
ps	Postscript format
text	Plain Text
tiff	TIFF graphics (file names only)

Additional formats may be available depending on your platform and the software version you're using. One version supports Microsoft Knowledge base files; some even allow you to define your own document types. To be sure of what you can and can't parse with WAISINDEX, check the latest version of your toolkit documentation.

Creating the database is an easy job. If WAISINDEX supports the types of files you want to include in your database, place them where they're going to reside and run WAISINDEX with command-line options that will give you the type of information you want in your database. What kind of command-line options? The two that are most often used are as follows:

Flag	Purpose
-d	Specifies a database name
-T	Informs WAISINDEX of the type of files being parsed

A sample command line might look something like the following:

% WAISINDEX -d /home/mydata -T HTML /files/*.HTML

This command line creates a database named mydata in the /home directory, setting the default type of file to parse to be HTML and indexing all the HTML files in the /files directory. Depending on the number of files you have to sort through, this process can take up a good deal of disk space and processor time; however, as an example, most indexes of less than 100 documents will be created in less than one minute.

Tools for Querying the Database

Creating your database was done for one reason: to allow people to search it. The indexed tables of data allow keyword searches to be sent from any user, to run through the appropriate mechanism, and to have results sent back. What's the "appropriate mechanism"? Well, you have a choice: WAISSEARCH or WAISQ.

WAISSEARCH

WAISSEARCH is the remote server for data. Like your HTTP server, FTP server, or anything else that listens to a port to provide feedback to requests sent to it, WAISSEARCH can be run in the background to process all those requests. On UNIX, it's started at the command line as a background process, whereas in NT and some other operating systems, it can be run as an automatic service. In either case, it's really concentrating on requests that come from some point outside your machine, rather than something local such as your own testing of the database.

WAISQ

WAISQ is the local search program. It doesn't sit and listen for information; it's executed from the command line and does all the searching right then and there. It's the easiest way to search through your database as a test for data that you know should be there. It's also the component that's used in locally executed scripts to grab input and bring it back to some program, such as a CGI-based script used with your Web server.

How to Query the Database

When a request comes in (locally or remotely), it has two basic components: the source to search through and what to search it for. Assuming the database can be accessed, the "seed words" of the query are checked against the source table of the database, and the output is generated. In the simplest case, a local query could be created to contain those two components, as in the following:

% WAISQ -d /home/mydata pancakes

In the preceding code, mydata is the database being specified (note that the -d option is common between indexing and searching for specifying a database), and pancakes is the keyword being searched for.

The searches performed aren't limited to that very generic format. Because WAIS supports Boolean operations, it could be a search for pancakes and syrup but not waffles or blueberries. This allows you to filter out more of the documents, returning only what's more relevant to your reason for searching. Another function of WAIS takes relevance one step further with relevance feedback, the capability to find a document that matches your parameters and send it back to ask for more documents like it.

Results of a Query

When the server or local program processes your query, all the items that match (up to any preset limit set by the server to keep processing time down) are returned to you. These pieces of information are also returned with one large benefit: ranking. If you receive 50 documents after your search, the first one in the list is the one that best matched your query. Normally, this is word-frequency based, which lends itself nicely to a little trick that people often use when listing their Web pages with a search site with this type of relevance ranking.

Because each occurrence of a particular word in the document is a match and results in an increase in the score, placing several hundred copies of that word inside the document in an invisible place will cause most searches to be very favorable toward ranking your document at the top. For instance, if you have an HTML document on recycling that you want to be on the top of the list, you can put the words recycle and recycling in a comment block at the bottom of the document, repeated several hundred times. Although you may mention it only twice in context, the search engine that parses sees all those occurrences and thinks your document must be the be-all and end-all of recycling information. Of course, this is the whole reason behind having different parsing formats: Using the correct parsing format, or setting up your own (as allowed by certain toolkits), allows you to strike comment fields and such from the ranking order. Of course, if you type it in very small letters on a non-comment portion of the document, that will make it by a comment-eliminating parse format.

When someone is trying to access the information you've placed in a WAIS database, three separate entities are trying to communicate: the client, your Web server, and your data's WAIS server. Communication between the client and your Web server is an easy two-way street and would most often be taken care of through the HTTP protocol. The level of difficulty in getting the client's request from your Web server, to your WAIS server's data, and back to your Web server for sending is another matter entirely. You need to establish a "gateway" between your Web server and the WAIS server, one that will do all the fetching and formatting for you whenever a request is made. The most common method of establishing this gateway is to use a CGI script (and it's important to note that this isn't necessarily the same as "write a CGI script"), but an option that's becoming more popular and more accessible is automatic integration between the HTTP server you use and the WAIS protocol.

Automated Server-WAIS Integration

If you're fortunate (or "foresightful") enough, you may have a server that supports automated integration of a WAIS database. These servers normally ship with a version of the WAIS toolkit for their platform to keep from even needing to hunt for the software in the first place. An example of this server-WAIS integration can be seen in Process Software's Purveyor server for NT and is illustrated in the following example.

You've created a WAIS database called "manual" that indexes a particular reference manual by page to provide a search function for your online HTML version of the manual. As a prototype, you create a simple form to provide a keyword search field. Later on, this can be integrated into a more elegant form after everything works to your satisfaction, but right now you're in a time crunch and just need to see it work. You start out with an HTML page, using the <ISINDEX> tag to provide the search field. It looks something like this:

<HTML>
This is a prototype search page.
<ISINDEX PROMPT="Enter words to search for here:">
</HTML>

Note

Defining searchable indexes in an HTML document relies on support for the <ISINDEX> tag to function properly. Most browsers today support <ISINDEX>, but not all. If you know that certain users who'll want access to your data will be using a package that doesn't support <ISINDEX> tags, many server-integrated search packages won't be right for your situation.

Now save the HTML document by placing it in the same directory as the WAIS database and by naming it manual.htm. When you view this document in a browser that supports the <ISINDEX> tag, it looks like figure 12.2.

With so many people out there using WAIS, you can turn to a number of places for more information on use, integration, and even future developments.

WWW Resources

The WAIS FAQ at Ohio State University is an excellent starting point and can be found at