User-agent: * Disallow: /en/search?q= Disallow: /suche?q= User-agent: * Disallow: /spraz/ Disallow: /*?state=wplan Disallow: /*&state=wplan Disallow: /en/course-catalogue? Disallow: /vorlesungs-verzeichnis? Disallow: /86260? Disallow: /index.php?id=86260&state=wplan Disallow: /118127? Disallow: /index.php?id=118127&state=wplan Disallow: /en/118127?state=wplan Disallow: /118129? Disallow: /en/timetables? Disallow: /index.php?id=1568 User-agent: msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) #url: http://search.msn.com/msnbot.htm #note: msnbots have changed their name recently Crawl-delay: 5 User-agent: MSNBot #url: http://www.msn.com/ #note: msn also accepts a delay Crawl-delay: 5 User-agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm) #url: http://search.msn.com/msnbot.htm #note: msnbots have changed their name recently Crawl-delay: 5 User-agent: Slurp #url: http://help.yahoo.com/help/us/ysearch/slurp/index.html #note: inktomi's "slurp" can read a minimum delay between hits Crawl-delay: 5 User-agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) Response by Siteimprove.com Disallow: / User-agent: Buck Disallow: / User-agent: PetalBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: drupact Disallow: / User-agent: PiplBot Disallow: / User-agent: SeznamBot Request-rate: 2/1m User-agent: SemrushBot Disallow: / User-agent: ZumBot Disallow: / User-agent: tracemyfile Disallow: / User-agent: dotbot Disallow: / User-agent: Mediatoolkitbot Disallow: / User-agent: AlphaSeoBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Cliqzbot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: Yahoo! Slurp Crawl-delay: 5 User-agent: Scrapy Disallow: / User-agent: daumoa Disallow: / User-agent: spbot Disallow: / User-agent: BLEXBot Disallow: / User-agent: sqlmap Disallow: / User-agent: ltx71 Disallow: / User-agent: Yanga WorldSearch Bot Disallow: / User-agent: jobo Disallow: / User-agent: EchO! Disallow: / User-agent: LinkChecker Disallow: / User-agent: Fish search Disallow: / User-agent: yacybot Disallow: / User-agent: Bloglines Disallow: / User-agent: Twiceler Disallow: / User-agent: Zealbot Disallow: / User-agent: VoilaBot Disallow: / User-agent: appie Disallow: / User-agent: sun4u Disallow: / User-agent: HTTrack Disallow: / User-agent: Nutch Disallow: / User-agent: Sisi Disallow: / User-agent: FyberSpider Disallow: / User-agent: girafabot Disallow: / User-agent: lmspider Disallow: / User-agent: NP Disallow: / User-agent: Robi Disallow: / User-agent: Webster Pro Disallow: / User-agent: Webster Disallow: / User-agent: Zeus Disallow: / User-agent: Gigabot Disallow: / User-agent: Scirus Disallow: / User-agent: Speedy Spider Disallow: / User-agent: PicoSearch Disallow: / User-agent: Plucker Disallow: / User-agent: DISCo Pump Disallow: / User-agent: Gulliver Disallow: / User-agent: vspider Disallow: / User-agent: EmailSiphon Disallow: / User-agent: Teleport Pro Disallow: / User-agent: Fetch Disallow: / User-agent: pamuk Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCapture Disallow: / User-agent: Mass Downloader Disallow: / User-agent: WebCopy Disallow: / User-agent: AWV0.8d Disallow: / User-agent: Crescent Internet ToolPak Disallow: / User-agent: JOC Web Spider Disallow: / User-agent: WebStripper Disallow: / User-agent: SiteSucker Disallow: / User-agent: Webdup Disallow: / User-agent: Scooter Disallow: / User-agent: Python-urllib Disallow: / User-agent: Python Disallow: / User-agent: Franklin Locator Disallow: / User-agent: CK-SillyDog Disallow: / User-agent: PocketHTTP Disallow: / User-agent: VoilaBot Disallow: / User-agent: Xaldon WebSpider Disallow: / User-agent: WebCapture Disallow: / User-agent: WebStripper Disallow: / User-agent: Java Disallow: / User-agent: WebReaper Disallow: / User-agent: TeragramWebcrawler Disallow: / User-agent: Vagabondo Disallow: / User-agent: nogoop-HttpClient Disallow: / User-agent: Baiduspider Disallow: / User-agent: MyOperaTB/1.0 Disallow: / User-agent: MyOperaTB Disallow: / User-agent: Yandex Disallow: / User-agent: vebidoobot Disallow: / User-agent: newspaper Disallow: /