Crawl errors happen when a search engine tries to access a page on your site and can’t. The page might not exist, the server might not respond, or something in your site’s configuration might be blocking access entirely. Whatever the cause, the result is the same: pages that Google can’t crawl don’t get indexed, and pages that don’t get indexed don’t rank. For sites with hundreds or thousands of URLs, crawl errors can quietly erode organic performance without anyone noticing until the traffic charts start trending downward.
At Gorilla Marketing, we build technical SEO strategies around crawlability and indexability as foundational priorities. If Googlebot can’t efficiently access and process your content, nothing else you do in SEO matters much. This guide walks through the most common crawl errors, how to find them, how to prioritize fixes, and how to prevent them from coming back.
Crawlability vs Indexability: What’s the Difference?
These two terms get used interchangeably, but they describe different stages in how Google processes your site.
Crawlability is whether Googlebot can access a URL. If the server returns an error, if robots.txt blocks the path, or if the page sits behind a login, it’s not crawlable. Googlebot never sees the content.
Indexability is whether Google can add a crawled page to its index. A page might be perfectly crawlable but still not indexed because of a noindex directive, a canonical tag pointing elsewhere, or because Google simply decided the content wasn’t worth indexing. Both matter, and they require different fixes.
Think of it this way: crawlability is about access. Indexability is about eligibility. You need both for a page to have any chance of ranking.
How Google Crawls Your Site
Googlebot discovers URLs through links, sitemaps, and previously known URLs. When it visits your site, it has a finite amount of resources to spend. That’s your crawl budget, and it’s determined by two factors: crawl rate limit (how fast Google can crawl without overloading your server) and crawl demand (how much Google wants to crawl based on perceived value and freshness).
For most sites under a few thousand pages, crawl budget isn’t a practical concern. Google can handle it. But for larger sites, or sites with significant technical debt, crawl budget becomes a real constraint. Every URL Googlebot spends time on is a URL it’s not spending time on somewhere else. If Googlebot is burning crawl budget on redirect chains, soft 404s, and parameter-bloated URLs, it has less capacity for the pages you actually want indexed.
The crawl stats report in Google Search Console shows you exactly how Googlebot is spending its budget on your site. Check the total crawl requests, average response time, and the breakdown of response codes. If you’re seeing a high percentage of non-200 responses, that’s crawl budget being wasted.
Finding Crawl Errors in Google Search Console
Google Search Console’s Page Indexing report (under the Indexing section) is your primary diagnostic tool. It shows every URL Google has attempted to process and the status of each one. The report groups URLs by reason, so you can see at a glance how many pages are excluded and why.
Key statuses to pay attention to:
Not found (404) – the URL returned a 404 error. The page doesn’t exist or was removed.
Server error (5xx) – the server failed to respond. Could be intermittent or persistent.
Soft 404 – the page returned a 200 status code but Google detected that the content looks like an error page. Empty pages, near-empty pages, and thin content often trigger this.
Blocked by robots.txt – your robots.txt file is preventing Googlebot from accessing the URL.
Crawled – currently not indexed – Google crawled the page but chose not to index it. This one requires more investigation because the cause could be content quality, duplicate content, or something else entirely.
Discovered – currently not indexed – Google knows the URL exists but hasn’t crawled it yet. This often indicates crawl budget constraints or low perceived priority.
Redirect error – the URL has a redirect problem, such as a chain, loop, or misconfigured redirect.
Blocked by noindex – the page has a noindex meta tag or X-Robots-Tag header telling Google not to index it.
Don’t panic about the total number of excluded pages. Some exclusions are intentional and correct. A noindex on your admin pages, a canonical tag consolidating duplicate parameter URLs, a robots.txt block on staging content: these are working as designed. The goal isn’t zero excluded pages. It’s ensuring that every page you want indexed is indexed, and that unintentional exclusions get fixed.
The Most Common Crawl Errors and How to Fix Them
404 Errors
A 404 means the URL doesn’t exist. This happens when pages get deleted without redirects, when URLs change during a site migration, or when external sites link to URLs that were never right in the first place.
Not all 404s are equal. A 404 on a page that had significant traffic, backlinks, or internal links pointing to it is a real problem. A 404 on a URL that nobody visits and nothing links to is noise. Google has stated repeatedly that 404s on URLs that shouldn’t exist aren’t harmful to your site’s overall SEO.
How to fix: For pages that were moved or replaced, implement 301 redirects to the most relevant current URL. For pages that were intentionally removed with no equivalent, let the 404 stand. For pages with significant backlink equity, redirect to the closest topical match. Use a tool like Screaming Frog to cross-reference your 404s with backlink data to identify which ones are actually worth fixing.
Soft 404s
Soft 404s are trickier. The server returns a 200 OK response, but the content of the page signals to Google that it’s effectively an error page. This happens with search results pages that return zero results, product pages for discontinued items that show a generic message, or category pages with no listed items.
How to fix: If the page has no content to show, return an actual 404 status code instead of a 200. If the page should exist but is empty due to a temporary issue (out-of-stock product, empty filtered view), either populate it with useful content or add a noindex directive until the content returns. Google needs a consistent signal: either this page has value, or it doesn’t.
5xx Server Errors
Server errors indicate that your server failed to process Googlebot’s request. A 500 error is a generic server failure. A 502 or 503 signals gateway or capacity issues. These can be intermittent (a brief outage, a spike in traffic) or persistent (a broken server configuration, a failing database connection).
How to fix: Check your server logs to identify when and why the errors are occurring. Intermittent 5xx errors during traffic spikes may mean your hosting can’t handle the load. Persistent 5xx errors on specific URLs often point to application-level bugs. If Googlebot consistently encounters server errors when crawling your site, it will reduce its crawl rate, which compounds the problem by slowing down indexing of your healthy pages.
Redirect Issues
Redirect chains occur when URL A redirects to URL B, which redirects to URL C, and possibly further. Each hop costs time and can lose PageRank. Google will follow multiple redirects in a chain, but that doesn’t mean you should rely on it. Long chains slow down crawling and dilute link equity.
Redirect loops happen when URL A redirects to URL B, which redirects back to URL A. Googlebot gives up, the page never gets crawled, and any user who hits that URL gets an error.
How to fix: Flatten redirect chains so that every old URL points directly to the final destination. Use Screaming Frog or a similar crawler to identify chains and loops across your site. After a migration, audit your redirect map within the first few weeks to catch chains before they become entrenched.
“Crawled – Currently Not Indexed”
This status means Google crawled the page but decided it wasn’t worth indexing. It’s one of the most frustrating statuses because the cause isn’t always obvious. Common reasons include thin content, duplicate content without proper canonicalization, low-quality pages, or Google simply determining the page doesn’t add enough value relative to what’s already in the index.
How to fix: Evaluate the content quality. Is the page genuinely useful to searchers? Does it offer something the existing indexed results don’t? If the content is thin, expand it. If it’s duplicating another page on your site, consolidate with a canonical tag or redirect. If it’s a legitimate page with substantial content that should be indexed, improving internal linking to the page can signal its importance to Google.
A Prioritization Framework for Crawl Errors
Here’s where most guides fall short. They list every error type and say “fix them all,” which isn’t practical when you’re looking at a Search Console report with 2,000 issues and limited dev time. You need a severity-based framework.
Tier 1: Fix immediately (revenue and ranking impact)
These errors are actively costing you traffic or revenue right now:
5xx errors on high-traffic pages. If your top landing pages are intermittently returning server errors, you’re losing both visitors and rankings. Fix the server-side issue before anything else.
404s on pages with significant backlinks or traffic. These represent lost link equity and lost visitors. 301 redirect them to the most relevant live page.
Redirect loops on important URLs. A loop means the page is completely inaccessible. No ambiguity, no partial fix. Resolve the loop.
Noindex accidentally applied to pages that should rank. This one is surprisingly common after CMS updates or theme changes. Audit your noindex directives regularly.
Tier 2: Fix this sprint (crawl efficiency and indexation)
These errors waste crawl budget or prevent pages from being indexed, but the impact is less immediate:
Redirect chains longer than two hops. Flatten them to single-hop 301s.
Soft 404s across templates. If a page template is generating soft 404s for an entire category (zero-result searches, empty filtered views), fix the template logic once.
“Crawled – currently not indexed” on important pages. Investigate and address the root cause, whether that’s content quality, internal linking, or canonicalization issues.
Blocked resources that prevent rendering. If your robots.txt is blocking CSS or JavaScript files that Googlebot needs to render your pages, you’re potentially hiding content from the index. Google needs to render your pages to understand them, especially with JavaScript-heavy sites.
Tier 3: Fix when capacity allows (cleanup and hygiene)
These are real issues, but they’re not dragging performance down in measurable ways:
404s on URLs with no backlinks and no traffic. Let them be or clean them up during routine maintenance.
Parameter URLs generating duplicate crawl requests. Handle with canonical tags or URL parameter configuration.
Legacy redirects from migrations years ago. Still working, just not optimal.
This framework gives your dev team clear priorities and prevents the common trap of spending weeks fixing low-impact 404s while server errors on your money pages go unaddressed.
Preventing Crawl Errors Before They Happen
Fixing crawl errors reactively is necessary. Preventing them is better. A few practices make a significant difference.
Maintain a clean XML sitemap
Your XML sitemap should only include pages you want indexed. Pages returning 404s, redirects, or noindex directives don’t belong in your sitemap. A sitemap full of non-indexable URLs sends mixed signals to Google and wastes crawl budget. We cover sitemap best practices in detail in our XML sitemaps guide – the key point here is that your sitemap should be a curated list, not an automated dump of every URL your CMS generates.
Configure robots.txt correctly
Your robots.txt file controls which parts of your site Googlebot can access. Misconfiguration is common, especially after site migrations or platform changes. A single overly broad disallow rule can block entire sections of your site from being crawled. Conversely, failing to block sections you don’t want crawled (staging environments, internal search results, admin areas) wastes crawl budget. We have a full breakdown in our robots.txt guide – for crawl error prevention, the main thing is to audit your robots.txt after any infrastructure change.
Use canonical tags consistently
Canonical tags tell Google which version of a page is the primary one when multiple URLs serve similar or identical content. Without them, Google has to guess, and it doesn’t always guess right. This leads to the wrong version being indexed, or neither version being indexed because Google can’t determine which to prioritize. Our canonical tags guide covers implementation in depth. For crawl error prevention, make sure every indexable page has a self-referencing canonical and that duplicate or parameter URLs canonical to the primary version.
Audit after every major change
Site migrations, CMS updates, theme changes, URL restructuring: any of these can introduce crawl errors at scale. Run a full-site crawl with Screaming Frog or a similar tool after every major change, and cross-reference the results with your previous crawl to catch new issues. Don’t wait for Google Search Console to surface problems weeks later. Proactive crawling catches issues while they’re still easy to fix.
Monitor with log file analysis
Server log files show you exactly what Googlebot is requesting and what responses it’s getting. This is the most granular view of how Google interacts with your site, and it reveals issues that Search Console doesn’t surface, like Googlebot hitting URLs that aren’t in your sitemap, or spending disproportionate time on low-value sections of your site. Log file analysis is the closest thing to reading Googlebot’s mind.
JavaScript Rendering and Crawl Errors
If your site relies on JavaScript to render content, you have an additional layer of complexity. Googlebot processes JavaScript in two phases: it first crawls the raw HTML, then queues the page for rendering with its Web Rendering Service (WRS). That rendering step isn’t instant. Pages can sit in the render queue, which delays indexing.
If your JavaScript fails to execute properly, or if critical resources are blocked, Googlebot may see an empty page. That’s a soft 404 at best and a complete indexing failure at worst. For JavaScript-heavy sites, use Google’s URL Inspection tool in Search Console to see the rendered output. Compare what Googlebot sees with what a user sees. If they don’t match, you have a rendering problem.
Mobile-first indexing adds another dimension. Google predominantly uses the mobile version of your content for indexing. If your mobile rendering is different from desktop, or if mobile-specific JavaScript errors prevent content from loading, those are the versions Google is evaluating. Test your mobile rendering specifically, not just desktop.
Internal Linking and Orphan Pages
Crawl errors aren’t always about broken pages. Sometimes the problem is that perfectly good pages aren’t getting crawled because nothing links to them. These are orphan pages: URLs that exist on your site but have no internal links pointing to them. If Googlebot can only discover pages by following links, and no links lead to a page, it may never find it.
Your internal linking structure is effectively a map of your site for Googlebot. Pages with many internal links get crawled more frequently. Pages with few or no internal links may be crawled rarely or not at all. A strong internal linking architecture ensures that your most important pages are well-connected and that Googlebot can reach every indexable page through a logical link path.
Screaming Frog’s crawl analysis can identify orphan pages by comparing your crawled URLs against your sitemap and server logs. Any URL that’s in your sitemap but wasn’t discovered during a link-based crawl is effectively an orphan. Fix these by adding contextual internal links from related pages.
When Crawl Errors Signal Bigger Problems
Sometimes a spike in crawl errors is a symptom of a larger issue. A sudden jump in 5xx errors might mean your server infrastructure is failing. A mass appearance of soft 404s might indicate a database connection issue causing pages to render without content. A wave of “discovered – currently not indexed” could mean Google is deprioritizing your site due to quality signals.
Look at the timing and patterns. Did the errors spike after a specific deployment? After a server migration? After a Google algorithm update? The pattern tells you whether the fix is technical (server, code, configuration) or strategic (content quality, site authority, relevance).
For sites where organic search drives significant revenue, monitoring crawl health shouldn’t be a quarterly review item. It should be a weekly check with alerts for sudden changes. Google Search Console can email you about critical indexing issues, and combining that with automated crawling tools gives you early warning before small issues become traffic-impacting problems.
Building a Crawl Health Routine That Sticks
Crawl errors aren’t a one-time cleanup. They accumulate constantly as content changes, URLs shift, and third-party integrations evolve. The sites that maintain strong organic performance treat crawl health like code quality: something you monitor continuously, not something you fix once and forget.
Build a monthly cadence: review the Page Indexing report in Search Console, run a Screaming Frog crawl, check server logs for anomalies, and validate that your sitemap and robots.txt are current. That routine takes a few hours per month and prevents the kind of technical debt that takes weeks to unwind.
If you’re looking at a Search Console report full of errors and aren’t sure where to start, or if your dev team needs a clear technical brief on what to fix and in what order, our technical SEO team works alongside engineering teams to diagnose crawl issues, prioritize by business impact, and build monitoring systems that catch regressions before they affect rankings.
