[BUG] WebCrawlerConnector accepts private/internal IP addresses (SSRF validation gap) #1709

Bug Description

In surfsense_backend/app/proprietary/web_crawler/connector.py, crawl_url() validates input URLs using only validators.url().

While validators.url() ensures syntactic RFC compliance, it does not validate the destination IP address. Consequently, URLs pointing to loopback, private subnets, link-local, or cloud metadata endpoints (such as http://169.254.169.254/latest/meta-data/ or http://127.0.0.1:8000) pass validation and are fetched by AsyncFetcher / StealthyFetcher.

In multi-user or Docker/cloud environments, this allows internal network requests from the backend container context.

Deployment Type

• SurfSense Cloud (hosted version)
• Self-hosted version

Steps to Reproduce

1. In Python:

import validators
# Both evaluate to True (bypassing the current check):
print(validators.url("http://169.254.169.254/latest/meta-data/"))  # True
print(validators.url("http://127.0.0.1:8000/health"))              # True

2. When configuring a crawl or invoking the web.crawl capability with an internal URL, WebCrawlerConnector.crawl_url accepts the URL at line 236:

if not validators.url(url):
    return CrawlOutcome(status=CrawlOutcomeStatus.FAILED, error=f"Invalid URL: {url}")

3. AsyncFetcher.get executes an outbound GET request to the internal host and returns extracted content.

Expected Behavior

The crawler should resolve hostnames to IP addresses and block private, loopback, link-local, and reserved IP ranges (e.g. 127.0.0.1, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.169.254).

Actual Behavior

Any syntactically valid URL is allowed through and fetched directly by the backend network stack.

Environment Information

• Operating System: Linux / Docker
• SurfSense Version: Commit b4d57e9b471891962a24bbd5c77620983509abfa (latest main branch)

Proposed Fix

Add a hostname resolution check that rejects private and reserved subnets:

import ipaddress
import socket
from urllib.parse import urlparse

def is_safe_url(url: str) -> bool:
    """Ensure target URL resolves only to public, non-reserved IP addresses."""
    parsed = urlparse(url)
    if parsed.scheme not in ("http", "https") or not parsed.hostname:
        return False
    try:
        ip = ipaddress.ip_address(socket.gethostbyname(parsed.hostname))
        return not (
            ip.is_private
            or ip.is_loopback
            or ip.is_link_local
            or ip.is_reserved
            or ip.is_multicast
        )
    except Exception:
        return False

Update crawl_url() in surfsense_backend/app/proprietary/web_crawler/connector.py:

if not validators.url(url) or not is_safe_url(url):
    return CrawlOutcome(
        status=CrawlOutcomeStatus.FAILED,
        error=f"Invalid or restricted URL: {url}",
        block_type=block_state["block_type"],
    )

Happy to open a PR with this fix if you'd like!

Checklist

• I have searched existing issues to ensure this is not a duplicate
• I have provided all the required information above