User-agent: ia_archiver Disallow: /perfectp/ Disallow: /how/test-pronunciation/ Disallow: /forum/ Disallow: /wiki/ Disallow: /misc/oldupdates.htm Disallow: /how/sm.htm Disallow: /about/contact.htm Disallow: /about/about.htm Disallow: /how/pronunc.htm Disallow: /how/pronunc-soundsipa.htm Disallow: /how/pronunc-learn-words.htm Disallow: /how/cobuild-review.htm Disallow: /other/myths-foraccent.htm Disallow: /how/srs-get.htm Allow: /about/ Allow: /how/ Allow: /learners/ Allow: /other/ Allow: /resources/ Allow: /terms/ Allow: /words/ Disallow: / User-agent: Slurp Crawl-delay: 20 User-agent: * Disallow: /wiki-base/ Disallow: /wiki/Special%3A* Disallow: /wiki/Talk%3A* Disallow: /wiki/MediaWiki%3A* Disallow: /wiki/Help%3A* Disallow: /wiki/User%3A* Disallow: /wiki/User_talk%3A* Disallow: /wiki/Special:* Disallow: /wiki/Talk:* Disallow: /wiki/MediaWiki:* Disallow: /wiki/Help:* Disallow: /wiki/User:* Disallow: /wiki/User_talk:* Disallow: /private/ Disallow: /wusage/ Disallow: /temp/ Disallow: /downloads/ Disallow: /learners/tomasz_szynalski.jpg Disallow: /learners/tomasz_szynalski-large.jpg Crawl-delay: 5