Questions and answers

The questions people ask about Skitter most, before they buy and while they crawl. If yours isn't here, the support page says how to reach me.

Before you start

What does Skitter need?

A Mac with macOS 26 Tahoe or later. Skitter is a Mac app, sold on the Mac App Store. There's no Windows or Linux version.

What's free?

Any crawl of up to 500 URLs, with everything Skitter does: every check, JavaScript rendering, every crawl setting, saved crawls, comparisons, exports, reports and sitemaps. A bigger site's crawl stops at 500 URLs and is saved, so you can look through what it found. What's free and what isn't has the full list.

What does a subscription add, and what does it cost?

A subscription carries a crawl past 500 URLs, to any size, and adds schedules and Shortcuts. It's $9.99 a month or $99.99 a year in the United States, and the App Store shows the price where you are. Both plans start with a week free.

How do I cancel, or get a refund?

Subscriptions are managed in your Apple Account settings in the App Store, where you can cancel at any time. Apple handles payments and refunds, at reportaproblem.apple.com. I don't see payment details, so I can't refund a purchase myself.

How big a site can Skitter crawl?

A crawl of 100,000 URLs is comfortable: lists and filters answer at once, and every count and report within a few seconds. 500,000 URLs is supported, with check counts taking about fifteen seconds. A crawl is stored on disk as it runs, so it uses a few hundred megabytes of memory whatever its size.

Crawling

Does Skitter render JavaScript?

Yes. Choose JavaScript rendered as the crawl's engine, and Skitter renders each page with WebKit, the engine that comes with macOS. A page's Source tab then shows the raw and rendered HTML side by side, and its Console tab shows what the browser reported. Rendering is slower than reading HTML, so it asks for fewer pages at a time.

Why does a site answer 403 Forbidden?

The site, or a firewall in front of it, is refusing the crawler. It may block unfamiliar user agents, or addresses that ask for many pages quickly. Try a slower crawl, or another user agent under How to ask in the crawl's settings. If it's your own site, let requests whose user agent contains Skitter through your firewall.

Why does a site answer 429 Too Many Requests?

The site is asking the crawler to slow down. Skitter does that by itself: each time a site answers 429, or 503 with Retry-After, it halves how many pages it asks for at once and waits before asking again. If a crawl still gets many 429s, lower Simultaneous requests and Max URLs per second in the crawl's Speed settings.

Crawling a Shopify store

Shopify limits how quickly one visitor can ask for pages, and answers 429 past that. Crawl a Shopify store with two simultaneous requests to start with; in my own tests, five at a time was already too many for one store. Expect each product to be reachable at more than one address (/products/… and /collections/…/products/…): Shopify points them all at one with a canonical, which the Canonicals checks show.

Crawling a site behind Cloudflare

Cloudflare's bot protection can answer with a challenge page instead of the page itself, which a crawler can't get past. The crawl then shows 403 or 503 answers, or the same short page everywhere. If it's your own site, add a rule in Cloudflare that lets your crawl through, for example by its user agent.

Can Skitter crawl a staging site behind a password?

Yes, when the site asks for a user name and password itself (basic or digest authentication). Enter them under How to ask ▸ Sign in as in the crawl's settings. The password is kept in your Keychain, never in the crawl. A sign-in form on a web page isn't supported.

Does Skitter obey robots.txt?

Yes, unless you tell it otherwise. Under How to ask ▸ robots.txt, a crawl can obey the rules, report what they block without obeying them, or ignore them.

Does a crawl stop when I close its window?

No. It carries on, with its progress in the Dock, and a stopped crawl can be resumed.

Results

What does "indexable" mean?

That nothing in the crawl prevents the page from being indexed. A page is non-indexable when it's blocked by robots.txt, marked noindex, canonicalised to another address, a redirect, or an error. Indexable doesn't mean a search engine has indexed it.

Why doesn't Skitter's count match Google or Search Console?

Skitter counts the addresses it could reach from where the crawl started, within the crawl's settings. Google indexes the pages it chooses, may know addresses that nothing on the site links to any more, and Search Console reports on its own schedule. The two answer different questions, so the counts rarely match.

Why doesn't it match another crawler?

Crawlers start from different defaults: whether images, scripts and external links count as addresses, whether robots.txt is obeyed, whether JavaScript is rendered, and how addresses are tidied. Compare the settings before comparing the numbers.

Why do two crawls of the same site differ?

The site changed, or it answered differently: a page that timed out, a 429, or content that changes on every visit. Crawl ▸ Compare with Previous Crawl shows exactly what changed between them.

Privacy

What does Skitter know about the sites I crawl?

Nothing reaches me. Your crawls, settings and passwords stay on your Mac, and the app has no analytics. The privacy policy has the details.

Skitter crawled my site. How do I slow it down or block it?

Skitter's crawler says how to recognise it, and how to slow it down or keep it out with robots.txt.

Help

Is there a command line?

Not in this version of Skitter.

How do I get help, or suggest something?

Email juye@juye.net. I aim to answer within 48 hours, and I read every suggestion.