![]() faulttoleranceproactive |
Introduction to writing spiders and agents |
![]() legalstuff |
![]() |
Fault Tolerance: Observe site policies | ||
1. | Stated usage policies | |
2. | Meta tags | |
<HEAD> | ||
<META NAME="ROBOTS" CONTENT="NOINDEX"> | ||
<HEAD> | ||
3. | Machine readable policies referenced in headers | |
![]() |