If you’ve ever tried running a technical SEO audit on a Shopify site using Screaming Frog (or similar tools like Sitebulb), you know the pain of 429 errors all too well. You start the crawl, everything looks promising for the first few hundred pages, and then bam, hundreds of “Too Many Requests” errors pop up, especially on product pages. The crawl stalls, your data is incomplete, and you’re left wondering how on earth you’re supposed to spot broken links, duplicate content, or indexing issues properly.
You’re not alone. This has been one of the biggest frustrations for Shopify merchants and SEO specialists since the dawn of time (or at least the dawn of Shopify), particularly on larger stores with tens of thousands of URLs where it’s difficult to even tell what percentage of the site has been crawled. The good news? In recent times, Shopify has made it much easier to get around this with a feature called Web Bot Auth, and there are still reliable fallback options if you need a quick fix. Let’s walk through why these errors happen and exactly how to sort them out so your next audit runs smoothly.
Why Shopify Throws 429 Errors at Your Crawler
Shopify runs on powerful infrastructure, but it has to protect itself from being overwhelmed. When a tool like Screaming Frog starts hammering the server with dozens (or hundreds) of requests per second, Shopify’s rate limiting kicks in and responds with a 429 Too Many Requests status. It’s basically the platform saying, “Slow down, you’re asking for too much too quickly!”
This is especially common on product-heavy stores because those pages make up the majority of the URLs. Without adjustments, your crawl might only get through a fraction of the site before it gets throttled.
A quick note on robots.txt too: Shopify’s default file is pretty sensible, it blocks junk like admin areas and overly filtered collections so search engines can focus on the good stuff. But if someone has made it overly strict, it can limit crawl budget for Googlebot and AI crawlers as well, which might slow down indexing and hurt rankings over time. The 429 issue itself is separate, though; it’s rate limiting, not robots.txt blocking.
Two Solid Ways to Beat the 429 Errors
Here are the most effective fixes based on what actually works in practice right now.
Option 1: Slow Down Your Crawl (Quick and No Admin Access Needed)
This is the easiest starting point and often all you need for smaller-to-medium stores.
In Screaming Frog, head to Configuration > Speed and tweak these settings:
- Max Threads: Drop it to 1 or 2 (default is usually higher).
- Max URLs per Second: Set to 1.
- Add a small Delay Between Requests if needed (1–2 seconds can help).
Your crawl will take longer, maybe a few hours instead of minutes, but it stays polite to the server and usually completes without a single 429. Many people find this alone solves the problem completely.
Pro tip: Use a proper User-Agent (don’t hide it or spoof it weirdly) so Shopify knows it’s a legitimate SEO tool.
Option 2: Authorise Your Crawler with Shopify’s Web Bot Auth (For Faster, Full-Speed Crawls)
Shopify rolled out Web Bot Auth using HTTP message signatures, and it’s a game-changer for serious audits on big stores. Once set up, your crawler is officially authorised, so rate limits don’t apply and you can run at normal (or even faster) speeds.
Here’s the step-by-step:
- Log into your Shopify admin.
- Go to Online Store > Preferences.
- Scroll to the Crawler access section and click Create signature.
- Give it a clear name (e.g. “Screaming Frog SEO Audit – Feb 2026”).
- Choose the relevant domain and set an expiration (up to 3 months max).
- Hit Create, then copy the three key values: Signature-Input, Signature, and Signature-Agent (usually set to “https://shopify.com“).
Now jump into Screaming Frog:
- Go to Configuration > Custom > HTTP Headers.
- Add the three headers exactly as provided (paste the values into the corresponding fields).
Save, restart the crawl, and you should sail through without throttling. Tools like Sitebulb and others support this too. Just remember signatures expire, so recreate them every few months as needed.
If you’re still seeing 429s after this, double-check the headers are being sent on every request, the domain matches, and the signature hasn’t expired.
How This Ties Into Google and AI Crawlers (and Your Rankings)
The 429 errors mostly hit third-party tools like Screaming Frog or Ahrefs – Googlebot has its own way of crawling and doesn’t use custom headers, so Web Bot Auth won’t help there directly. But if your site is constantly slow or error-prone because of other issues, Google might dial back its crawling too, which can lead to sluggish indexing and rankings.
A clean, complete crawl lets you fix those underlying problems early: optimise page speed, tidy up redirects, improve internal linking, and submit proper sitemaps. All of that helps Google (and emerging AI crawlers) discover and rank your content more reliably.
Get Expert Eyes on It If Needed
Crawling Shopify doesn’t have to be a headache anymore. Whether you slow things down for a gentle crawl or set up Web Bot Auth for full power, you can get the complete data you need to make smart SEO decisions.
But let’s be honest, sometimes the crawl is just the start. You uncover a bunch of technical issues, dev tweaks, or content gaps, and suddenly you’re juggling a to-do list that’s bigger than expected. If auditing your Shopify store feels overwhelming or you’re not seeing the rankings lift you’d hoped for, chatting with specialists can save a lot of time and frustration.
At Trafficon Digital, we live and breathe eCommerce SEO, including Shopify and WordPress SEO services. We handle these crawls every week, interpret the results accurately, and implement fixes that actually move the needle. If you’d like help with your next audit or want to chat about your site’s SEO performance, drop us a line; we’re always happy to lend a hand.
Here’s to stress-free crawls and stronger search visibility!
