Skitter's crawler
Skitter is a Mac app that people use to check websites for broken links, redirects, missing titles and other problems. If Skitter visited your site, someone was checking it from their own Mac, usually their own site or one they look after. Nothing it reads is sent to me.
How to recognise it
When the person crawling chooses Skitter as the user agent, it says so:
Mozilla/5.0 (compatible; Skitter/1.0; Macintosh; +https://skitterapp.com/bot)
The number is the version of the app. The person can also choose to look like Safari or Googlebot, or type a user agent of their own, so not every visit from Skitter says Skitter.
What it does
- It obeys robots.txt unless the person crawling turns that off. Their own crawls are their responsibility (see the terms of use).
- It reads robots.txt rules for
Skitter, then for*. The name is matched case-insensitively. - It follows
noindexandnofollowin robots meta tags andX-Robots-Tagheaders named forskitter, as well as those for every robot. - It asks for five pages at a time unless told otherwise, and slows down when a server answers 429 or 503 with
Retry-After.
How to slow it down or block it
To keep Skitter out of part of your site, or all of it:
User-agent: Skitter
Disallow: /private/
User-agent: Skitter
Disallow: /
A crawl that has been told to ignore robots.txt won't see these rules. To stop that too, block requests whose user agent contains Skitter at your server or firewall.
Contact
If Skitter is causing a problem on your site, email juye@juye.net.