Web crawler information

You may be wondering about “sp_auditbot” in your weblogs or web statistics and be curious why it is visiting your site. The sp_auditbot is the indexing bot that is used by the website audit tool in SEOprofiler.

If you see this bot in your web logs, you usually have an SEOprofiler account for that website. If you don’t have an SEOprofiler account, someone else has tested your site with the audit tool in SEOprofiler. To block sp_auditbot from crawling your site, please read below.

How can I block the sp_auditbot?

We will not index anything you would like to remain private. All you have to do is tell us. How? By using the official Robots exclusion standard. Example:

User-agent: sp_auditbot
Disallow: /

You can find detailed instructions on how to block web crawlers on Wikipedia and on the robots.txt page.

Make sure that you don’t inadvertently block other crawlers that you don’t want to block, for example, the Googlebot.

Why shouldn’t I block the sp_auditbot?

The website audit tool will only visit your web pages if you create a project for that website. If you don’t want to use the website audit tool, just change the settings in your SEOprofiler project. You don’t have to block the bot. Just disable the website audit tool in your account.

How can I verify spbot is really spbot?

Other web crawlers can spoof the sp_auditbot user agent to make them seem legitimate. They seem to be coming from us but they don’t. We use Amazon Web Services for our crawlers, so you can verify the IP addresses that sp_auditbot uses here.

Feel free to contact us: terms _AT_ seoprofiler -DOT- com

SEOprofiler users