CHAPTER 15 - Generating HTML Documents in Real Time

HyperText Markup Language (HTML) lets you publish text and graphics in a platform-independent way. Using HTML, you can easily, via embedded links, weave a world-full of sites together.

In this chapter, you examine static and dynamic HTML, concentrating on the latter. Dynamic, or real-time, HTML extends the viability of the Web far beyond its original conception.

You learn what makes real-time HTML tick and how to produce it in a variety of ways. Specifically, this chapter provides

• An overview of static HTML
• An introduction to real-time HTML
• The benefits of real-time HTML
• Methods of generating real-time HTML, including scheduled jobs, regular CGI or SSI, Client Pull, and Server Push
• An example of dynamically redirecting a browser to a static HTML page appropriate for that browser
• An example of Server Push
• Server performance considerations

Need to review the complete works of Mark Twain? Want to find the address of a manufacturer in Taiwan? Need the phone number for the White House? Ever wondered how to spell floccinaucinihilipilificatrix? Or what it means? (Yes, that's a real word. You won't find it in any dictionary except the Old English Dictionary., though, so put away your Webster's Collegiate.)

The answers are only as far away as your favorite search engine. These types of references are perfectly suited to the Web. They seldom, if ever, need revision; after they're written and thrown on a page, other sites can establish links to them, search engines can catalog them, and you can find them-today, tomorrow, next week, or next year. Because the markup language used to create these pages is HTML and the content of the pages is static (relatively unchanging), such pages are called static HTML.

But what if you want to know the stock prices-not 10 hours ago or 10 days ago, but right now? What if you want to know the arrival time of American Airlines Flight 101? What if you need to know the ambient temperature in Brisbane as of 30 seconds ago?

In these cases, static documents just won't do. Not even if a diligent, never-sleeping Webmaster does his level best to keep the documents updated. For these sorts of applications, you need real-time, or dynamic, HTML.

All CGI-generated HTML is technically "real-time" in that it's generated on the fly, right when it's needed. In data processing circles, however, the term refers more to the data itself than the production thereof.

Therefore, a CGI program that talks to a hardware port and retrieves the current temperature and then generates HTML to report it would be considered real-time. A CGI program that looks up your birthday in a database wouldn't.

In this chapter, I don't worry too much about the technical definitions. I call all CGI programs that produce time-sensitive or user-sensitive output "real-time." This includes uses such as the following:

• Current temperature Quote of the day
• Current time and date Network or server statistics
• Election returns Package delivery status
• Stock market data Animations and other special effects
• Page hit-count for a home page Browser-specific pages

Benefits of Real-Time HTML

The prime, and most immediately apparent, benefit of real-time HTML is that the information is fresh. Getting the stock market report from yesterday's closing is one thing; finding the value of a specific stock right this minute is something else altogether. The information has different value to the consumer. People pay for up-to-the-minute information.

Another, somewhat less obvious, benefit is that real-time HTML can make your pages seem livelier. For example, in the next chapter you examine a page counter and a random-quote generator. You can put them together on a page to produce output like this:

• First visit: "You're the 1st visitor to be amused by this page."
• Second visit: "You're the 2nd visitor to be flabbergasted by this page."
• Third visit: "You're the 3rd visitor to be terrified by this page."

And so on. Granted, this particular example is rather trivial. Many readers may not even notice that the wording changes each time, and those who do won't have their lives, careers, or religion changed by it. But this example should give you an idea of the sorts of pages you can make by using real-time document generation.

Methods of Generating Real-Time HTML

The following are the four main methods of generating dynamic pages:

• Scheduled Jobs
• Regular CGI or SSI
• Client Pull
• Server Push

In the following sections, you tackle them in order.

Scheduled Jobs

A scheduled job is a batch file, shell script, or other program that runs at a regular interval. These jobs usually run in the background-that is, invisibly and independent of the foreground task-and may run once a month, once a day, or once a minute. The interval is up to you. A special case is the program that runs continuously (called a daemon in the UNIX world, and a service in the Windows NT world), spending most of its time asleep, and waking up only periodically to accomplish some task. Usually, though, background jobs are scheduled. They run at the appointed time, do their jobs, and quit, only to repeat at the next scheduled time.

The method of scheduling varies from operating system to operating system. In UNIX, you find the cron utility most appropriate. Under Windows NT, the AT command makes the most sense.

Scheduled jobs are useful for information that changes infrequently but regularly. A quote-of-the-day program is probably the best example. You don't need to invoke a CGI program to retrieve or regenerate a program that changes only once a day. It's far better to write a program that updates your HTML at midnight and then let the page get retrieved normally.

Regular CGI or SSI

For page counters and similar programs, either CGI or SSI (see the next chapter for examples of SSI) make the most sense. The kind of information being generated is what drives your choice. Because a page count changes only when a page is retrieved, updating it then makes sense. A scheduled job is clearly inadequate for up-to-the-moment data, and the remaining methods-Client Pull and Server Push-are inappropriate because you don't want a continuous update.

A trivial, but nonetheless useful, example of using CGI to provide dynamic HTML is a CGI program that redirects the browser to a static page appropriate for that browser. For this example, assume that you want to provide different pages for each of the following browser types: Netscape, Microsoft Internet Explorer, and Lynx. Any browser that can't be identified as one of these three gets redirected to a generic page.

ByAgent is a complete working sample of using CGI to provide a dynamic response. You should be able to compile it for any platform. You can find the source, plus sample HTML files and a compiled executable for the 32-bit Windows NT/Windows 95 environment.

Compile the code (as shown in listing 15.6) and name it byagent.exe. Put the compiled executable in your CGI-BIN directory. If you're using a 32-bit Windows environment, you can skip the compile step.

To test this program, you need to create a number of static HTML files. The first will be used to demonstrate the others. Call it default.htm.

Listing 15.1 default.htm: HTML to Demonstrate ByAgent
<html>
<head><title>ByAgent</title></head>
<body>
<h1>ByAgent Test Page</h1>
This page demonstrates the ByAgent CGI program.  Click <a href="/cgi-bin/byagent.exe?">here</a> to test.
</body>
</html>

As you can see, this code is fairly straightforward. If your CGI-BIN directory is called something else, correct the link in the preceding code.

Now you can create four individual pages: one for Netscape, called netscape.html (see listing 15.2); one for Lynx, called lynx.html (see listing 15.3); one for Microsoft Internet Explorer, called msie.html (see listing 15.4); and one for everyone else, called generic.html (see listing 15.5).

Listing 15.2 netscape.html: Target Page for Netscape Browsers
<html>
<head><title>ByAgent</title></head>
<body>
<h1>ByAgent</h1>
Congratulations! You got to this page because your browser identified itself as a Netscape (or compatible) browser.
</body>
</html>
Listing 15.3 lynx.html: Target Page for Lynx Browsers
<html>
<head><title>ByAgent</title></head>
<body>
<h1>ByAgent</h1>
Congratulations! You got to this page because your browser identified itself as a Lynx (or compatible) browser.
</body>
</html>
Listing 15.4 msie.html: Target Page for MSIE Browsers
<html>
<head><title>ByAgent</title></head>
<body>
<h1>ByAgent</h1>
Congratulations! You got to this page because your browser identified itself as a Microsoft Internet Explorer (or compatible) browser.
</body>
</html>
Listing 15.5 generic.html: Target Page for Generic Browsers
<html>
<head><title>ByAgent</title></head>
<body>
<h1>ByAgent</h1>
Congratulations! You got to this page because your browser identified itself as a something other than Netscape, Lynx, or Microsoft Internet Explorer.
</body>
</html>

Put these files together in a directory, and load default.htm into your browser. Click the test link. You should see the page corresponding to your browser. Listing 15.6 shows the actual code to accomplish the redirection.

Listing 15.6 byagent.c: Source Code for ByAgent CGI Program
// BYAGENT.C
// This program demonstrates how to redirect
// a browser to a page that matches the browser.
// It depends on the browser's self-identification,
// so a browser that lies can get the wrong page.
// In general, most programs that claim to be
// "Mozilla" are either Netscape, fully compatible
// with Netscape, or Microsoft Internet Explorer.
// The special case of MSIE can be identified
// because although it says "Mozilla," it also
// says "MSIE."

#include <windows.h>
#include <string.h>
#include <stdio.h>

void main() {

     // First declare our variables.
     // We'll use three pointers and a character
     // array.  The pointers are UserAgent, a
     // pointer to the CGI environment variable
     // HTTP_USER_AGENT; Referer, a pointer to
     // the CGI environment variable
     // HTTP_REFERER; and p, a generic pointer
     // used for string manipulation.  The
     // remaining variable, szNewPage, is where
     // we build the URL of the page to which
     // the browser gets redirected.

     char     *UserAgent;
     char     *Referer;
     char     *p;
     char     szNewPage[128];

     // Turn buffering off for stdout
     setvbuf(stdout,NULL,_IONBF,0);

     // Get the HTTP_REFERER, so we know our directory
     Referer = getenv("HTTP_REFERER");

     // Get the user-agent, so we know which pagename to
// supply
     UserAgent = getenv("HTTP_USER_AGENT");

     // If either user agent or http referer not available,
// die here
     if ((Referer==NULL) | (UserAgent==NULL)) {
          printf("Content-type:  text/html\n\n"
            "<html>\n"
            "<head><title>ByAgent</title></head>\n"
            "<body>\n"
             "<h1>Pick your browser</h1>\n"
            "ByAgent could not find either the "
            "HTTP_REFERER or the HTTP_USER_AGENT "
            "environment variable.  "
            "Please pick your browser from this list:\n"
             "<ul>\n"
             "<li><a href=\"generic.html\">Generic</a>\n"
             "<li><a href=\"lynx.html\">Lynx</a>\n"
             "<li><a href=\"msie.html\">Microsoft</a>\n"
             "<li><a href=\"netscape.html\">Netscape</a>\n"
             "</ul>\n"
             "</body>\n"
             "</html>"
             );
          return;
     }

     // This program assumes that the browser-specific pages
     // are in the same directory as the page calling this
     // program.  Therefore, we'll use the HTTP_REFERER to
     // get our URL, then strip the HTTP_REFERER's page
// name, and add the proper browser-specific page name
// to the end.

     // First, copy the HTTP_REFERER value to szNewPage, so
     // we have something to work on.
     strcpy(szNewPage,Referer);

     // Find the last forward slash in the URL.  This is
     // the separator between the directory and the page
     // name.
     p = strrchr(szNewPage,'/');

     // If we found no forward slash, assume some sort of
     // weird server and hope a relative path will work by
     // chopping off the entire URL.
     if (p==NULL) p = szNewPage;

     // Mark the end of the string, so we can concatenate
     // to it from that point on.
     *p = '\0';

     // Convert to lower-case so we can do more efficient
     // searches.
     _strlwr(UserAgent);

     // We are now ready to output a redirection header.
     // This header tells the browser to go elsewhere
     // for its next page.  A redirection header is
     // nothing more than a standard content type
     // followed by "Location: " and an URL.  The
     // content type is separated from the redirection
     // by a single newline; the entire header is
     // terminated by a blank line (two newlines).

     // If user agent is Microsoft Internet Explorer,
     // redirect to msie.html
     if (strstr(UserAgent,"msie")) {
          printf("Location: %s/msie.html\n\n",szNewPage);
          return;
     }

     // If user agent is Lynx,
     // redirect to lynx.html
     if (strstr(UserAgent,"lynx")) {
          printf("Location: %s/lynx.html\n\n",szNewPage);
          return;
     }

     // If user agent is Netscape,
     // redirect to netscape.html
     if (strstr(UserAgent,"mozilla")) {
          printf("Location: %s/netscape.html\n\n",szNewPage);
          return;
     }

     // If none of the above,
     // use generic.html
     printf("Location: %s/generic.html\n\n",szNewPage);
     return;
}

As you can see, the preceding code is fairly simple. The comments far outweigh the lines of code. The only tricky bits to this program are (a) remembering to format the redirection header correctly, and (b) remembering that Microsoft Internet Explorer claims to be "Mozilla" (Netscape) if you don't look carefully.

In your own program, you may want to incorporate some mechanism to allow the secondary pages to live in a different directory, or even on a different server, just by changing the Location information. You may also consider generating the correct HTML on the fly rather than redirect the browser to an existing static page. Now that you know how to identify the browser and do redirection, your imagination is the only limit.

Client Pull

Client Pull is a Netscape enhancement. Several other browsers now support Client Pull, but you should be careful when writing your HTML to include options for browsers that can't deal with it.

In typical browsing, a user clicks a link and retrieves a document. With Client Pull, that document comes back with extra instructions-directives to reload the page or to go to another URL altogether.

Client Pull works via the META HTTP-EQUIV tag, which must be part of the HTML header (that is, before any text or graphics are displayed). When the browser sees the META tag, it interprets the contents as an HTTP header. Because HTTP headers already support automatic refresh and redirection, not much magic is involved at all. Normally, the server or CGI program is responsible for sending the HTTP headers. Netscape's clever idea was to allow additional HTTP headers inside a document.

Say you have a Web page that reports election returns. A background process of some sort reads the precinct numbers from a Reuters connection (why not?) and once every 10 seconds rewrites your Web page with the current data. The client can hit the reload button every ten seconds to see the new data, but you want to make that process automatic. Listing 15.7 shows how to do it.

Listing 15.7 default.htm: Demonstration of Client Pull
<html>
<head>
<META HTTP-EQUIV="Refresh" CONTENT="10">
<title>Election Returns</title>
</head>
<body>
<h1>Election Returns</h1>
This document refreshes itself once every ten seconds.  Sit back and watch!
     ...
</body>
</html>

Note the META HTTP-EQUIV line. This line causes the browser to refresh the page once every 10 seconds. Of course, for this example to be useful, you need to have some other process updating the page in the background, but this example works-it will reload the page once every 10 seconds.

Why once every 10 seconds? Because each time it fetches the document, the browser sees the instruction to load it again ten seconds later. The instruction is a "one-shot" instruction. It doesn't tell the browser to load the page every ten seconds from now until doomsday; it just says to load the page again ten seconds from now.

As you saw earlier in this chapter in the section "Methods of Generating Real-Time HTML," not everything needs to be generated on the fly. Documents that are updated regularly and served as static documents are often called near real-time, because the information is fresh but the document itself is static. Often, CGI is used to update the document (rather than create the document in real time). This allows the document to reflect changes immediately, but avoids the overhead of running a CGI program every time a browser fetches the document.

Dynamic HTML can be a lot of fun and can be extraordinarily useful at times. However, it doesn't come without cost.

The first consideration is for caching proxy servers. If your page includes a page count or a random quotation or a Server Push animation, it can't be cached. Defeating caching isn't necessarily an evil-you wouldn't want your up-to-the-second stock market quotes to be cached, for instance-but it can create unnecessary network traffic.

If you visit the Usenet groups regularly, you see a recurring theme of experienced old hackers venting their spleens at newbies who chew up bandwidth for no reasonable purpose. The range of opinion you find goes from calm, rational argumentation, to wild, impassioned screeds. Some go so far as to say that any CGI program is evil and that page hit-counters are the devil's own spawn.

In a book with the sole purpose of teaching you how to write your own CGI scripts, you won't find much support for the extremists. The network is there to be used. Like any limited resource, it should be used wisely rather than wastefully. The problem is in determining what's wise. If you keep your high-traffic pages static, you'll make everyone except the true Internet curmudgeons happy. Of course, if you're a Java developer, all bets are off. The new ways of using the Web are completely incompatible with caching from the start.

The second thing to consider is that CGI programs tax the Web server. For each retrieval that calls a CGI program, the server must prepare environment variables, validate security, launch the script, and pipe the results back to the caller. If a hundred scripts are executing simultaneously, the server may become overburdened. Even if the server has sufficient resources to cope, the overall server throughput will suffer.

Server Push puts more of a strain on the system than almost any other type of dynamic HTML, because the script continues executing (consuming processor cycles and memory) theoretically forever. Just a few of these scripts running at the same time can bring an otherwise capable server to its knees. They have a high level of traffic and resource consumption for relatively little gain.

There are no hard and fast rules. As with any system, you must balance performance against cost, and capacity against throughput.