# robotstxt.org/ # 管理・設定の画面・API・計測用のリダイレクトはクロールさせない(Config\Seo::ROBOTS_DISALLOW_PATHS と同じ。#115)。 # /auth・/tracking は noindex を読ませるため塞がない。公開ページ・商品詳細・sitemap は塞がない。 # クローラーは自分の名前のグループがあると * のグループを読まないため、名前付きのグループにも同じ Disallow を書く。 Sitemap: https://price2alert.com/sitemap.xml User-agent: * Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: Slurp Crawl-delay: 10 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: bingbot Crawl-delay: 1200 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: msnbot Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: Yeti Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: Baiduspider Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: Baiduspider+ Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: BaiduImagespider Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: BaiduMobaider Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: MLBot Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: AhrefsBot Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway User-agent: Ezooms Crawl-delay: 10000 Disallow: /admin Disallow: /setting Disallow: /webservice Disallow: /widget Disallow: /event Disallow: /gateway