# ============================ # Apple Crawlers # ============================ User-agent: Applebot Crawl-delay: 10 Disallow: User-agent: Applebot-Extended Disallow: / # ============================ # Ahrefs Crawlers # ============================ User-agent: AhrefsBot Crawl-delay: 10 Disallow: User-agent: AhrefsSiteAudit Disallow: / # ============================ # Bing / Microsoft Crawlers # ============================ User-agent: Bingbot Crawl-delay: 10 Disallow: User-agent: BingPreview Disallow: / # ============================ # Yandex Crawlers # ============================ User-agent: Yandex Crawl-delay: 10 Disallow: User-agent: YandexBot Crawl-delay: 10 Disallow: User-agent: YandexImages Disallow: / # ============================ # PetalBot (Huawei) # ============================ User-agent: PetalBot Crawl-delay: 10 Disallow: # ============================ # OpenAI Crawlers (no delay support) # ============================ User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # ============================ # Amazon Crawlers (no delay support) # ============================ User-agent: Amazonbot Disallow: / User-agent: Amzn-SearchBot Disallow: / User-agent: Amzn-User Disallow: / # ============================ # Default Rule for All Other Bots # ============================ User-agent: * Crawl-delay: 10 Disallow: Disallow: /content/css Disallow: /content/images Disallow: /content/font Disallow: /content/OwlCarousel Disallow: /views/