Apache OpenOffice (AOO) Bugzilla – Issue 118940
Tune robots.txt for openoffice.org website
Last modified: 2017-05-20 10:45:16 UTC
We currently have: ============== User-agent: daumoa Disallow: / User-Agent: Sosospider Disallow: / User-Agent: Snoopy Disallow: / User-Agent: * Crawl-delay: 5 Disallow: /nonav/ Disallow: /welcome/ Disallow: /white_papers/ ============== Some of this doesn't make sense. For example, there is nothing at /nonav, /welcome seems to have a redirect issue that we should just fix. And /white-papers/ has reasonable content that we should not be hiding from index spiders.
When we have done our first browser tests for MSIE versions with "www.browsershots.org" I've updated the robots.txt already. IMHO this task is done and I would close this issue as fixed.
Reset the assignee to the default "issues@openoffice.apache.org".