User-agent: * Disallow: /users/ #Comments... Disallow: /stats/ #Comments... Disallow: /temp/ #Comments... Disallow: /multimedia/ #Comments... Disallow: /photolibrary/ #Comments... #Disallow: /startrek/images/ #Comments... Disallow: /models/ #Comments... Disallow: /cgi-bin/ #Comments... Disallow: /software/ #Comments... Disallow: *paz666_* #This is bait to catch bad robots. Disallow: *wshow9_* #This is bait to catch bad robots. Disallow: /*.gif$ Disallow: /*.jpg$ Disallow: /*.jpeg$ Disallow: /*.png$ User-Agent: Googlebot Allow: /cgi-bin/startrek/screencaps.pl Allow: /cgi-bin/startrek/screencap_large.pl Allow: /startrek/ Allow: /cgi-bin/startrek/screencaps_comments.pl Allow: /cgi-bin/startrek/screencaps_search.pl Allow: /cgi-bin/startrek/transcript_search.pl User-agent: vscooter User-agent: DittoSpyder User-Agent: Googlebot-Image User-Agent: ia_archiver User-agent: IconSurf Disallow: / Allow: /startrek/images/episodes/ Allow: /cgi-bin/startrek/screencaps.pl Allow: /cgi-bin/startrek/screencap_large.pl