CheckerrBot

Checkerrbot is our crawler daemon - our robot visiting websites instead of us automatically. We want to be decent internet-citizens, so the User-Agent of Checkerrbot includes the string "checkerrbot" and a link to this website.

Should the way Checkerrbot crawls your website cause your any issues, please get in touch at thaltercheckerr.org.

Because Checkerrbot aids you in determining errors on your website, it would make little sense to respect the contents of your website's robots.txt file, as human visitors don't do so. It would simply surpress errors your visitors DO encounter continously, so you couldn't fix them.