User-agent: * Disallow: / Crawl-delay: 120 Disallow: /cgi-bin/