Search Security

How do I protect my search from bots and abusive queries?​

Most websites see some amount of "search noise": bots sending bursts of requests, users pasting long strings into the search box, or automated tools probing random characters and URLs.

For example, imagine your site normally gets a few hundred searches a day, but you suddenly see thousands of requests per hour from the same IP address — or queries like <<<<<<<<<<< and long garbled strings that clearly do not come from real visitors. In cases like this, it makes sense to add a few guardrails.

• If you can identify offending IPs directly — block them, cap how many queries an IP can run per hour, or exclude your own team's testing traffic from logs. See IP Settings and Query Limits for the specific fields.
• If the abuse is query-shaped rather than IP-shaped — for example, overly long queries, queries full of special characters, or queries containing links — the Spam Filter can decline those requests automatically. See Spam Filter for the specific fields.

Protecting your site from bots​

Site Search 360 can help you rate limit traffic and decline suspicious queries, but it cannot stop bots from reaching your website in the first place. If you are dealing with heavy bot traffic, consider adding protection upstream — before requests ever reach your search endpoints.

Effective bot mitigation works best as a layered effort. Think of it as a funnel: the earlier you identify and block bot traffic, the less impact it has on your query quota and your users' experience. Site Search 360 sits at the end of that funnel. The measures below are outside of Site Search 360 but represent the highest-impact steps you can take.

CDN and Web Application Firewall (WAF)​

A CDN or WAF sits in front of your website and can block the majority of bot traffic before it reaches your search implementation. This is the most effective single step you can take.

Cloudflare is the most widely used option, with meaningful bot protection available even on its free plan:

• Bot Fight Mode (Free) — automatically detects and challenges known bad bots.
• Rate Limiting Rules (Free and above) — cap requests per IP within a time window, targeted at your search path (e.g. paths or query strings containing q= or /search).
• Super Bot Fight Mode (Pro and above) — uses machine learning to distinguish verified crawlers from malicious bots.
• Bot Management (Enterprise) — assigns every request a bot score and lets you define rules per score range.

Further reading: Bot Fight Mode · Rate Limiting Rules · Super Bot Fight Mode · Bot Management

If you are hosted on a major cloud platform, each offers its own WAF with bot control capabilities:

Further reading: AWS WAF Bot Control · Azure Web Application Firewall · Google Cloud Armor

Server-side rate limiting​

If you manage your own server, you can add rate limiting directly at the server level targeting your search endpoint. This complements the built-in Max. Queries per IP per Hour setting in Site Search 360, catching excessive traffic before it reaches our service.

For example, the following Nginx configuration limits each IP to 10 search requests per minute with a short burst allowance:

limit_req_zone $binary_remote_addr zone=search:10m rate=10r/m;

location /search {
    limit_req zone=search burst=20 nodelay;
    limit_req_status 429;
}

Additional server-level measures worth considering:

• User-Agent filtering — block requests from known bot signatures such as curl, python-requests, or scrapy.
• Request header validation — real browsers consistently send headers like Accept, Accept-Language, and Referer; many bots do not.
• Fail2ban — automatically bans IPs that trigger an unusual number of errors within a short time window.

Further reading: Nginx rate limiting module · Rate Limiting with Nginx · Fail2ban

CAPTCHA and challenge-response​

CAPTCHA operates at the website level, not at the search endpoint itself. Tools like reCAPTCHA v3 run passively on the page, scoring each visitor's behaviour. If a visitor's score falls below your threshold — indicating likely bot activity — you can block or challenge them before a search request is ever fired.

Apply challenges selectively. Adding a visible CAPTCHA to every search interaction creates unnecessary friction for real users. The recommended approach is to trigger a challenge only when suspicious behaviour is detected, such as an unusually high query rate from a single session or a low trust score returned by your WAF.

Restricting access to search endpoints​

This addresses a potential security risk for intranet and extranet projects: if your search endpoints are publicly accessible, anyone who knows — or can guess — a valid siteId could query the Search and Suggestion endpoints directly and retrieve indexed results. This means your content may be exposed through the API even if your website itself is not publicly accessible.

Protect Search Endpoints mitigates this risk by requiring authorization on all search and suggestion requests — either an API key or an allowed host name list. See Protect Search Endpoints for the specific fields.