Introduction to writing spiders and agents
Fault Tolerance: Observe site policies
   1. Stated usage policies
       
   2. Meta tags
    <HEAD>
        <META NAME="ROBOTS" CONTENT="NOINDEX">
    <HEAD>
       
   3. Machine readable policies referenced in headers