SitelensBot visited your site. Here’s what it is.

SitelensBot is the crawler behind Sitelens, an SEO audit service. It fetches public pages so that site owners, and the people they authorise, can find and fix technical SEO problems.

User-agent string

Mozilla/5.0 (compatible; SitelensBot/1.0; +https://sitelens.dev/bot)

robots.txt token SitelensBot

Concurrency ~5 requests

JavaScript Not executed

01 —What it does and doesn’t do

Does

Fetches public pages over HTTP and HTTPS, the way a search engine does
Identifies itself in every request with the SitelensBot user-agent
Reads robots.txt and obeys it by default during site audits
Reads your XML sitemaps to discover pages
Uses around 5 concurrent requests per audit and stops at the plan’s page limit
Extracts SEO fields only: status, redirects, titles, meta descriptions, headings, canonicals, word counts and links

Does not

Submit forms, sign in, or fill in fields
Execute JavaScript or load ads and analytics tags
Request private or internal network addresses
Store full page HTML
Probe for vulnerabilities or try to reach non-public areas
Sell or publish what it finds; results go only to the user who ran the audit

02 —Why it’s visiting

A Sitelens user added your site, or a page from it, to an audit. Our Terms and Acceptable Use Policy allow full-site audits only of websites the user owns or is authorised to audit. Single-page checks fetch just the one URL a user asks about.

Users can switch off robots.txt only for sites they own or are authorised to audit. If you see SitelensBot ignoring your robots.txt and you haven’t authorised anyone to audit your site, please tell us.

03 —Block or limit it with robots.txt

Block SitelensBot from your whole siteSitelensBot stops crawling as soon as it reads this rule.

User-agent: SitelensBot
Disallow: /

Keep it out of specific sectionsUseful for large archives, faceted search or staging paths.

User-agent: SitelensBot
Disallow: /search
Disallow: /tag/
Disallow: /*?sort=

Allow only part of your siteThe most specific (longest) matching rule wins, as with Google.

User-agent: SitelensBot
Disallow: /
Allow: /blog/

04 —Report a problem

If SitelensBot is causing trouble on your site, or you think someone is using Sitelens to crawl it without authorisation, contact us. Include your domain, roughly when you saw the requests, and any relevant log lines. We can throttle or block crawls of your site straight away.

Report abuse [email protected]