# amazoo.co.il — Bee1 baseline, added 2026-07-27 # Reason: meta-externalagent/1.1 was crawling the faceted-filter space on /shop/ # at ~6 req/s (95% of all traffic), burning ~3 CPU cores on vps3 and creating a # WooCommerce session per add-to-cart hit. Nginx enforces this too # (xcloud-conf/amazoo.co.il/server/zz-bee1-filter-trap.conf); robots.txt is the polite layer. # Meta's AI-training crawler. Not the ads/link-preview scraper (facebookexternalhit), # so blocking this does NOT affect Facebook/Instagram ad traffic or link previews. User-agent: meta-externalagent Disallow: / User-agent: meta-externalads Disallow: / User-agent: meta-webindexer Disallow: / User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # Faceted filter URLs: infinite combinations, duplicate content, zero index value. Disallow: /*?filter_ Disallow: /*&filter_ # Cart actions are never worth crawling. Disallow: /*add-to-cart= Sitemap: https://amazoo.co.il/sitemap_index.xml