# robots.txt for https://asiavirtualsolutions.com # Purpose: SEO friendly, WooCommerce safe, and LLM friendly. # Allows public content to be crawled while blocking low-value and transactional URLs. # Last updated: 20 September 2026 # # 20 Sept 2026: added /affiliate-area/ and /logout/ (plus their */ wildcard # forms) to both the main group and the AI crawler group. Both returned 200 and # were crawlable. They are member/session URLs with no search or AI value. # The six AI reference files listed below were aligned to this file on the same # date - if they ever disagree, THIS file is authoritative. Sitemap: https://asiavirtualsolutions.com/sitemap_index.xml # LLM discovery files # https://asiavirtualsolutions.com/llms.txt # https://asiavirtualsolutions.com/llms-full.txt # https://asiavirtualsolutions.com/llms.json # https://asiavirtualsolutions.com/llms.yaml # AI usage policy: https://asiavirtualsolutions.com/ai.txt # Agent capabilities: https://asiavirtualsolutions.com/agents.json # ------------------------------------------------------------ # WHY SO MANY RULES START WITH * # ------------------------------------------------------------ # TranslatePress serves 21 translated language subdirectories: # ar bn de es fa fr hi id it ja ko ms nl pt ru th tl tr ur vi zh # A rule written as "/cart/" matches ONLY the English path, so /de/cart/, # /th/cart/ etc stayed crawlable. "*/cart/" matches both. The plain form is # kept alongside it for the few old crawlers that ignore wildcards. # Rules that already begin with /* (parameter rules) need no change - the * # in them also matches a language prefix. # ------------------------------------------------------------ # MAIN RULES - every bot NOT named in a group further down # ------------------------------------------------------------ User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # WordPress low-value / duplicate / utility URLs Disallow: */trackback/ Disallow: /xmlrpc.php Disallow: /cgi-bin/ Disallow: */comments/ Disallow: */search/ Disallow: /*?s= Disallow: /*&s= Disallow: /*?replytocom= Disallow: /*&replytocom= Disallow: /*?orderby= Disallow: /*&orderby= # NOTE (Sept 2026): /feed/ and /wp-json/ are deliberately NOT blocked. # Feeds: blocking them stopped Google seeing they are worthless, so it indexed # 20 translated feed URLs with no content. Feeds now send X-Robots-Tag: noindex # (Code Snippets #155, active) so Google can crawl them and drop them permanently. # wp-json: blocking the REST API gives no SEO benefit and can hinder rendering. # Affiliate redirect links (ThirstyAffiliates cloaked links) # Keeps crawlers off the redirect URLs so they are not followed or indexed. # /tool/ is the custom link prefix set in ThirstyAffiliates. Disallow: /tool/ Disallow: */tool/ # WooCommerce / account / transactional pages # /checkout/order-pay/ and /checkout/order-received/ sit under /checkout/, # so they are already covered by the two checkout rules below. Disallow: /cart/ Disallow: */cart/ Disallow: /checkout/ Disallow: */checkout/ Disallow: /my-account/ Disallow: */my-account/ Disallow: /affiliate-area/ Disallow: */affiliate-area/ Disallow: /logout/ Disallow: */logout/ Disallow: /*?add-to-cart= Disallow: /*&add-to-cart= Disallow: /*?wc-ajax= Disallow: /*&wc-ajax= Disallow: /*?coupon_code= Disallow: /*&coupon_code= # WooCommerce filter / duplicate parameter URLs Disallow: /*?filter_ Disallow: /*&filter_ Disallow: /*?filter= Disallow: /*&filter= Disallow: /*?min_price= Disallow: /*&min_price= Disallow: /*?max_price= Disallow: /*&max_price= Disallow: /*?rating_filter= Disallow: /*&rating_filter= # Optional: only enable if tag, author, and date archives have no SEO value for AVS # Disallow: */tag/ # Disallow: */author/ # Disallow: */date/ # ------------------------------------------------------------ # Ayzeo GEO + SEO: AI crawler policy # ------------------------------------------------------------ # AI and LLM crawlers are explicitly welcome on all real content. # IMPORTANT - do not simplify this to a bare "Allow: /". robots.txt uses # most-specific-group matching: a bot named below reads ONLY this group and # ignores "User-agent: *" entirely. The disallow lines must therefore be # repeated here, or these crawlers would crawl cart, checkout, my-account, # search results and every filter parameter URL. # # amazonbot moved here from the blocked list (Sept 2026): it respects # robots.txt and now feeds Amazon Rufus as well as Alexa, so blocking it # only cost AI visibility. It is not a backlink scraper. User-agent: gptbot User-agent: oai-searchbot User-agent: chatgpt-user User-agent: claudebot User-agent: claude-web User-agent: anthropic-ai User-agent: google-extended User-agent: perplexitybot User-agent: perplexity-user User-agent: applebot-extended User-agent: amazonbot User-agent: ccbot User-agent: cohere-ai User-agent: omgilibot User-agent: diffbot User-agent: facebookbot User-agent: meta-externalagent User-agent: duckassistbot Allow: / Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: */trackback/ Disallow: /xmlrpc.php Disallow: /cgi-bin/ Disallow: */comments/ Disallow: */search/ Disallow: /*?s= Disallow: /*&s= Disallow: /*?replytocom= Disallow: /*&replytocom= Disallow: /*?orderby= Disallow: /*&orderby= Disallow: /tool/ Disallow: */tool/ Disallow: /cart/ Disallow: */cart/ Disallow: /checkout/ Disallow: */checkout/ Disallow: /my-account/ Disallow: */my-account/ Disallow: /affiliate-area/ Disallow: */affiliate-area/ Disallow: /logout/ Disallow: */logout/ Disallow: /*?add-to-cart= Disallow: /*&add-to-cart= Disallow: /*?wc-ajax= Disallow: /*&wc-ajax= Disallow: /*?coupon_code= Disallow: /*&coupon_code= Disallow: /*?filter_ Disallow: /*&filter_ Disallow: /*?filter= Disallow: /*&filter= Disallow: /*?min_price= Disallow: /*&min_price= Disallow: /*?max_price= Disallow: /*&max_price= Disallow: /*?rating_filter= Disallow: /*&rating_filter= # ------------------------------------------------------------ # BLOCKED BOTS - scrapers / no SEO value # ------------------------------------------------------------ # NOTE (Sept 2026): AhrefsBot and SemrushBot were unblocked so those platforms # can crawl asiavirtualsolutions.com and return full site-audit and on-page data. # Amazonbot was moved to the AI crawler group above. # Bytespider stays blocked: very aggressive crawl rate, widely reported to # disregard robots.txt, and it feeds Doubao / TikTok search (domestic Chinese # market) which is not where AVS customers are. If it ever shows up heavily in # the server logs, block it at server or Cloudflare level - robots.txt may not # be enough on its own. User-agent: MJ12bot User-agent: DotBot User-agent: Bytespider User-agent: PetalBot User-agent: BLEXBot Disallow: /