SightCrawl

SightCrawl is the news crawler operated by FireSight. It reads publicly available news articles to extract headlines, publication dates and text for news-signal analysis.

How it behaves

• Identifies itself with the user agent SightCrawl/0.1 (+https://sightcrawl.firesight.ai/).
• Honours robots.txt, including Crawl-delay.
• Makes at most one request at a time to a site, with a pause between requests.
• Does not log in, solve captchas or get around paywalls or bot protection. If a site blocks it, it stops.

Opting out

To stop SightCrawl from fetching your site, add this to your robots.txt:

User-agent: SightCrawl
Disallow: /

Questions or concerns: contact FireSight via firesight.ai.