Sevenfixes

Our crawler

SevenfixesBot

If you found this page in your server logs: SevenfixesBot fetched a few pages of your site because someone who manages it in Google Search Console asked Sevenfixes to check it. Here is what it does and how to control it.

How to recognise it

Every request carries this user agent:

SevenfixesBot/1.0 (+https://sevenfixes.com/bot)

In robots.txt it answers to the name SevenfixesBot. It never pretends to be a browser or another bot.

Which sites it visits

  • Only sites whose owner connected Google Search Console to Sevenfixes, and only after Search Console shows that their account has access to the site. Before that, we send no request to the site at all.
  • Only URLs on the site's own domain that are listed in its sitemaps, plus robots.txt and the sitemaps themselves. It does not follow links across the web.
  • A small sample: a few pages of each page type (product, category, article and so on), not the whole site.

How fast it goes

  • One request at a time by default, with a pause of 2 seconds between requests. The pause is never shorter than 1 second, and never more than 2 requests run at once on one site.
  • It slows down when your server answers 429 (too many requests) or 503 (unavailable), and it stops when it keeps meeting challenge pages.
  • It never tries to get around a block: no fake user agents, no solving of challenges.

How to block it

Add this to your robots.txt and SevenfixesBot will not fetch your pages:

User-agent: SevenfixesBot
Disallow: /

You can also block only some paths with Disallow: /path/. If you are a Sevenfixes customer, blocking the bot means the fixes that need to see your pages cannot be found or checked; the parts built from Search Console still work.

How to let it through a firewall

If your site sits behind Cloudflare or another WAF that challenges bots, choose one of these:

  • A secret header (best). In your site's settings in Sevenfixes, set a header name and a secret value. We send it with every request; add a firewall rule that skips the challenge when the header matches. The value is stored encrypted.
  • Our server's IP address. Write to [email protected] for the current address and allow it in your firewall.
  • The user agent. Possible, but anyone can copy a user agent, so we do not recommend it on its own.

Something wrong?

If SevenfixesBot misbehaves on your site, or you did not ask for it, write to [email protected] with your domain and the time of the requests. We will look into it and can stop all fetching of a site at once.