User-agent: Jobbot Disallow: / User-agent: barkrowler Disallow: / User-agent: dotbot Disallow: / User-agent: Yandex Disallow: / # prohibits crawling for the entire site User-agent: AwarioRssBot User-agent: AwarioSmartBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: DotBot Disallow: / User-agent: JobBot Disallow: / User-agent: rogerbot Disallow: / User-agent: AccompanyBot Disallow: / User-agent: Adsbot Disallow: / User-agent: SeznamBot Disallow: / User-agent: * Allow: / Crawl-Delay: 5