Crawler allowlisting

Some sites block crawlers by default. To ensure reliable indexing, allow AskVio crawler traffic through WAF/CDN/security rules.

• Allow known crawler user-agent(s).
• Allow crawler source IP ranges if your firewall needs static rules.
• Permit sitemap and content routes without challenge pages.
• Avoid bot challenges (CAPTCHA/JS challenge) on pages that must be indexed.