The SemwiserSiteAudit crawler
What the crawler is, what it sends, and how to keep it out.
SemwiserSiteAudit is the crawler behind SEO, a website check in Semwiser. It visits a site only when a Semwiser customer starts a check of that site, or on a repeat schedule that customer set up. It reads its pages the way a search engine would, and reports technical and on-page problems back to that customer.
It reads robots.txt before it crawls and follows the rules the file sets for all crawlers (User-agent: *). Pages are crawled from our crawling provider’s servers. Semwiser’s own servers fetch only your robots.txt and llms.txt, and send SemwiserSiteAudit/1.0 when they do.
How it identifies itself
A check that does not render JavaScript sends SemwiserSiteAudit/1.0 and a link to this page in the User-Agent header of its requests. The customer who starts the check chooses one of two forms:
- Mobile: Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Mobile Safari/537.36 (compatible; SemwiserSiteAudit/1.0; +https://support.semwiser.com/site-health/site-audit-crawler)
- Desktop: Mozilla/5.0 (compatible; SemwiserSiteAudit/1.0; +https://support.semwiser.com/site-health/site-audit-crawler)
A check that renders JavaScript loads your pages in a browser, which may send its own User-Agent header instead of the one above, so those visits may not show SemwiserSiteAudit in your logs.
How fast it crawls
By default it waits one second between pages. The customer can slow it to one page every two seconds, or have it wait as long as your robots.txt Crawl-delay asks, up to 60 seconds between pages.
Keeping it out
To keep crawlers out of part of your site, add a Disallow rule for that path in the User-agent: * group of your robots.txt. To keep every crawler out of the whole site:
- User-agent: *
- Disallow: /
To keep only this crawler out of the whole site, write a group for it instead. A check is not started when a User-agent: SemwiserSiteAudit group disallows the address it would start from:
- User-agent: SemwiserSiteAudit
- Disallow: /
If you think it crawled your site in a way it should not have, write to support with the time of the visit and the address that was crawled.