1998 Innovation Collection
| Benefits | Importance | Originality | Success | Difficulty | Long Summary The goal of the Mars Pathfinder web site was to enable as many people as possible from around the world to participate in viewing the Mars images in near real time. Because of the anticipated large number of people wishing to access the site, a typical web site would not be able to handle the load. The primary constraint was one of bandwidth; there are only two T3 lines coming into JPL with a combined bandwidth capacity of 110 megabits per second. Therefore, even if JPL acquired a powerful supercomputer to host the Pathfinder web site, it still would not be able to handle the load due to the bandwidth constraint. For this reason the Pathfinder site had to be replicated to many identical sites around the world to ensure that the images would simultaneously be available on the Internet to millions of people. The process is called mirroring and although it was well established as a method of spreading load, JPL applied it more extensively than any organization has in the past. What was unique about the Pathfinder mirror sites was that they had to be updated while potentially millions of people were accessing them. For this reason standard commercial off-the-shelf products could not be relied upon to function properly under these conditions. Most commercial mirroring software compares the modifications dates of the files forming the master site and the remote mirror site to determine which files have been added or changed on the master site. However, searching the entire directory system of the remote site takes time and is prone to failure when the site is already heavily loaded with access requests. For this reason JPL wrote custom software to determined which files had been added or modified on the master site and then sent those specific updated files to the remote mirror sites. The updated files were first consolidated into a tar file ( a tar file is roughly equivalent to a pkzip file under a common windows or PC DOS environment ) to ensure that all of the files arrived at the mirror site before the updating process began. The use of UNIX tar files also had the advantage that they are cross platform, being supported under Microsoft NT and the Macintosh. When Pathfinder landed on July 4, 1997, JPL had 20 mirror sites in place around the world in order to handle the load. From any mirror site it was possible to get a list of all of the other mirror sites. Therefore, if one site was inactive or had poor response time, another mirror site could easily be selected. As a result, the collective network of Pathfinder mirror sites was extremely robust and had phenomenal performance given the number of people who were accessing the distributed site. In the first 4 days after landing the Pathfinder site collectively logged over 100 million hits world wide. The biggest day occurred on July 8th with an estimated 46.9 million hits to the Pathfinder web site. In the month of July the site received over 520 million hits world wide from over 120 countries. It was truly a global event in which people were able to collectively participate in an historic event in real time through their own home computers. During the month of July thousands of congratulatory e-mails were sent to JPL from around the world. In terms of space exploration no event has better captured the attention of the public since the Apollo Moon landings. As an Internet event it was the biggest in history. Prior to Pathfinder the most any web site had received in a day was 22 million hits which occurred during the 1996 Olympic games in Atlanta. In summary, the Pathfinder web site demonstrated that the Internet is a new global information distribution mechanism with the capability to transmit digital information with greater ease and speed than any other form of mass media.
Benefits The primary benefit of the Pathfinder web site was to increase people’s awareness of our solar system and our new technology which enables us to collectively experience its wonders. The Pathfinder site has also improved people’s feelings about our government. In the past there has been a general belief that the government takes the taxpayers money, gains insight into the world around us, and then neglects to share their findings with the people. With the Pathfinder site, so much information was immediately made available to the public that charges of NASA withholding data were not taken seriously by the majority. In fact, in a number of editorials, the Pathfinder project was hailed as a new chapter of openness in government. In terms of rapid data release, Pathfinder has shown what is possible and set the standard by which all future NASA planetary missions will be judged.
Importance The Internet was the key element that enabled the rapid transmission of the digital data from Mars to home computers. Just five years ago, it would not have been possible to achieve such remarkable results. With the exception of the custom code written at JPL, no new technology was developed to deploy the Pathfinder web site. The custom code consisted of a simple utility that recursively searched the directory system of the main site and logged the new or modified files off to a relational database. The files logged in the relational database were then sent to a consolidation area under UNIX which consisted of an empty directory tree. Once the files were placed in their proper location within the directory tree, the entire file structure under the consolidation area was wrapped up into a tar file and then sent via FTP to the remote mirror sites. At this point the tar file was unraveled with a simple UNIX command, updating the remote mirror site.
Originality To our knowledge, the Pathfinder Web site is the best application of mirroring technology currently in existence on the Internet. No other web site has applied it as extensively or with greater success. The Pathfinder mission itself was ground breaking in many areas. It was the least expensive mission to ever land on another planet. It had a development time of only 4 years from conception to completion, which is between a half to one third of the development time of past planetary missions. It was the first planetary mission ever to use an airbag landing system. Perhaps most significant to the general public, it was the first Mars lander to deploy a rover capable of exploring the surrounding terrain. The success of the small team of people who made the Pathfinder site possible is similar to individual success stories that made the Pathfinder mission possible. If you analyze the success of the Pathfinder mission you will see that it is composed of small groups of people doing whatever was necessary to make sure their component of the system worked. The entire Pathfinder team was composed of only 50 people. The people who designed and built the spacecraft also flew the spacecraft. Everyone on the flight team knew their part of the spacecraft inside and out, since they built it. The same was true of the small team who built and operated the Pathfinder web site: Robert Anderson, David Dubov, and Kirk Goodall. The Pathfinder web site was started by Robert Anderson in March of 1996 well before the launch of the Pathfinder spacecraft from Cape Kennedy in December of 1996. Initially the site existed on a 486 a computer running Windows NT. At this point in the project the number of visitors to the site was quite low and bandwidth was not a concern. Although in theory the dual processor 486 could handle a million hits per day, Anderson became concerned that it would soon be outstripped and that a far more substantial system would be needed to handle the hits around the time of launch and certainly before the time of Landing. Since the entire Pathfinder project was on a shoestring budget, there were no funds available to purchase a powerful UNIX machine capable of handling even a fraction of the anticipated load. In April of 1996 Anderson began networking within JPL and seeking the advice of experts for a solution to what would certainly be a bottleneck around the time of landing. Also, in April of 1996 Kirk Goodall, a programmer at JPL, independently developed an application that acquired pictures of his television at home and sent them into JPL for display on an internal JPL web site. Goodall developed this application in order to promote his desire to place real-time video feeds of the launch of the Mars missions onto the Internet. It was also at this time that the European Space Agency (CNES) recommended to Cheick Diarra of the Mars Educational Outreach Office that the Pathfinder project consider mirroring as a method of handling the anticipated Internet traffic during landing. Cheick Diarra was aware of Kirk Goodall’s interest in Internet coverage of the Mars missions and therefore placed him in contact with Robert Anderson of the Pathfinder project. At this point in the project there were no funds available to explore mirroring technology. However, Goodall recognized the need and took up the task of investigating mirroring technology in his free time during the evenings and on the weekends. His first task was to determine what had to be done to the site before it could be mirrored. In June of 1996 Anderson obtained additional assistance from David Dubov who improved the overall look and feel of the site. Goodall examined Dubov’s HTML code and determined that all of the absolute links had to be converted to relative links, or else the mirror sites would refer back to JPL for HTML content, defeating the purpose of the mirror sites. During August of 1996 Goodall and Dubov worked together in converting the HTML code in order to make the site mirrorable. Goodall also began evaluating mirroring software and the special conditions that would exist around the time of landing. It was estimated that the eventual network of mirror sites would have to handle somewhere between 20 and 30 million hits per day around the time of landing. Goodall soon realized that no commercially available mirroring software was up to the task and began developing his own code under NT and UNIX. He also began the task of finding organizations that were willing serve as mirror sites for Pathfinder. In October of 1996 two NASA research centers were selected with high bandwidth connections: AMES, and Kennedy Space Center. By November of 1996 the first mirror site was setup at AMES and Goodall’s mirroring technology was successfully demonstrated. It was at this point that $15,000 was set aside within a public outreach account in order to offset some of the costs of the mirroring effort. However, this could not cover all of the expenses, and like many of the engineers working on Pathfinder, Goodall continued to donate time to the project. In February of 1997 it was determined that the bandwidth available to the NASA centers would not be sufficient, and therefore mirror sites at other government organizations would be needed. After analyzing the resources available within the government sector, Goodall selected the most powerful network available, the National Science Foundation backbone. He approached all five NSF super computer centers, and convinced them to provide JPL with mirror sites. The NSF sites consisting of the Cornell Theory Center, the San Diego Supercomputer Center, the Pittsburgh Supercomputer Center, the National Center for Atmospheric Research, and the National Center for Supercomputer Applications (NCSA). These sites combined with the three NASA centers had a theoretical capacity of 30 million hits per day. It was thought that this would be sufficient until May of 1997 when it was discovered that the NSF backbone would not be open to commercial Internet traffic during landing. It was at this point that Goodall, Anderson, and Diarra began exploring the possibility of working with commercial firms in securing additional bandwidth. However, because JPL is part of the Federal Government, there are strict rules in place constraining the manner in which JPL could enter into teaming arrangements with the commercial sector. Goodall was placed in contacted with JPL’s legal department, obtained a boiler plate Memorandum of Understanding (MOU), and rewrote the MOU to account for the constraints that JPL was under. This MOU became the legal document that enabled JPL to set up mirror sites at commercial firms. Ten days before landing Goodall managed to setup high bandwidth mirror sites at Silicon Graphics, Sun Microsystems, CompuServe, and AT&T World Net, giving a combined capacity of 80 million hits per day. With these additional sites there was undoubtedly sufficient capacity to handle the anticipated load. As landing approached, Goodall and Dubov became the core team responsible for updating the master site at JPL and replicating its content to the 20 mirror sites world wide. Since JPL could not monetarily compensate the system administrators at the mirror sites, Goodall agreed to handle the mechanics of setting up the mirror site and updating it. The mirror sites were only required to provide a machine with a high bandwidth connection to the Internet. During time of the Mars Pathfinder landing, Dubov was responsible for generating all HTML web content on the master site at JPL and Goodall was responsible for making this content available to the world through the network of mirror sites. In generating the HMTL content, Dubov was assisted by Nathan Bridges, Richard Pavlovsky, and Stephanie Zeluck who gathered images approved for release and wrote the corresponding captions that were incorporated into the web content. In addition to the mirroring technology, Goodall and Dobov developed real time Internet applications that displayed engineering data as it came down from the spacecraft. This was the first time engineering data feeds had been made available during each phase of the mission. The engineering feeds were very popular and have since been incorporated into the current mission, Mars Global Surveyor, with even greater success. Not only does the general public benefit from these feeds, but also the engineers and scientists responsible for the spacecraft. The Internet feeds are currently being used by remote science teams to monitor their instruments.
Success The success of the Pathfinder Web Site can be gauged by the number of hits and number of people who have accessed the site. By December 31, 1997 an estimated 722 million hits had been logged, which corresponds to about 20 million visits to the Pathfinder site. During the period around landing, it became clear to the television networks that the images were available on the Internet and that numerous mirror sites has been setup to handle the demand. At this point the media began to actively promote the Internet component of the Mars landing and this was the primary reason the web site was so successful. The media promoted the Internet and the Internet encouraged people to stay tuned into the media to see what images were about to be released onto the Internet. A symbiotic relationship developed between the media and the Internet that was largely responsible for the huge number of hits received. Difficulty There were three primary obstacles that had to be overcome in order to make the network of Pathfinder mirror sites possible:
[ Search Again | Innovation Network | Copyright (c) 1998 ] |