Quick answer

To fix crawl errors and indexing issues, check the affected URL's HTTP status first, then robots.txt and noindex, canonical choice, internal links and JavaScript rendering. Use Google Search Console URL Inspection to confirm what Google saw. Different errors need different fixes; requesting indexing is not a substitute for correcting a blocked or broken page.

Common crawl errors and what they mean

A crawl error means a search bot had trouble requesting or processing a URL. An indexing issue means Google has not included the preferred page in search. One can cause the other, but they are not the same. Start by grouping affected URLs by what the server actually returned.

404 Not FoundThe page is missing. Restore it if it should exist, 301 redirect it to a real replacement, or keep a genuine 404 if it was intentionally removed.
Soft 404The page loads but looks empty, irrelevant or like an error. Return the correct status and useful content instead of a misleading 200.
500 / 502 / 503Server failure or temporary outage. Check hosting, application logs and CDN settings before changing content.
403 / blocked requestAccess denied. Confirm firewalls or security tools are not incorrectly denying legitimate crawlers.
Redirect errorLoop, chain or redirect to the wrong URL. Resolve the path to one relevant final destination.

Google provides a detailed crawler troubleshooting guide covering availability, crawl efficiency and error pages.

Start with the preferred URL, not the symptom

When a page disappears from search, it is easy to jump straight to keywords or backlinks. First confirm that the URL you want indexed is actually the URL your website is presenting as the preferred version. Open it in a browser, inspect the status code, check the canonical tag and make sure it is not redirecting somewhere unexpected.

A healthy important page will normally return a 200 status, be allowed to crawl, avoid an accidental noindex, and point its canonical to the correct preferred URL. Google documents these foundations in its Search Essentials and canonical guidance.

Check robots rules and noindex separately

robots.txt controls crawling. A meta robots noindex controls indexing when the crawler can access the page. They solve different problems. Blocking a page in robots.txt can stop Google from seeing a noindex directive placed on that page.

Review the site's robots file, page-level meta robots and any HTTP X-Robots-Tag headers. Google explains the role of robots rules in its robots.txt documentation.

Find canonical conflicts before creating new content

Duplicate or near-duplicate URLs can send mixed signals. Common examples include HTTP and HTTPS versions, www and non-www versions, old .html URLs, parameter URLs, print pages and duplicate category paths.

Check whether the canonical matches the URL you actually want indexed. Also check redirects, sitemap entries and internal links. These signals should point in the same direction. Google's canonicalisation guidance explains why several consistent signals are stronger than one isolated tag.

Internal links tell crawlers which pages matter

A page can be technically indexable and still be difficult to discover if nothing important links to it. Search engines follow links, and users rely on the same paths. Important service, category and content pages should be linked from relevant pages with normal HTML anchors.

Look for orphan pages, JavaScript-only navigation, generic anchor text and deep click paths. Google recommends crawlable <a href> links in its link best-practices documentation.

“Links are not crawlable”: how to fix the problem

A link should usually be a real <a href="/target-page"> element with a working destination. A button with only an onclick handler, an empty href, a hash-only placeholder or a link created only after a user action may be unreliable for discovery. Keep essential service and article links in the HTML, use clear link text and make sure the target URL returns the intended page.

For example, link from an SEO services page to a topical mapping guide using a phrase that explains the destination. A technical SEO article should link to a relevant website migration checklist when the reader needs a redirect plan.

Use the sitemap as a clean list of preferred URLs

Your XML sitemap should contain canonical, indexable URLs you actually want in search. It is not a place to dump every URL your CMS can generate. Remove redirects, 404s, blocked URLs and duplicate variants. Keep the sitemap current after launches, migrations and content changes.

Check rendering, server errors and JavaScript

If the HTML response is thin and important content only appears after heavy JavaScript runs, crawling and rendering become more fragile. Also review intermittent 5xx errors, timeouts and CDN problems. A page that works in your browser today may still have availability problems that affect crawlers.

For JavaScript sites, make sure essential content and internal links are available reliably. For static sites, keep the delivery path simple and avoid unnecessary client-side rendering for primary content.

Use Search Console to verify the diagnosis

Search Console's URL Inspection tool can show whether Google knows the URL, which canonical it selected and whether indexing is allowed. Use the report as evidence, not as a replacement for checking the website itself. Compare what Google reports with the live HTML, status code and internal-link path.

A practical crawl and indexing priority order

1. Availability200 status, no server errors, no redirect loop.
2. AccessRobots rules allow important crawling.
3. IndexabilityNo accidental noindex or conflicting header.
4. CanonicalThe preferred URL is consistent across signals.
5. DiscoveryRelevant internal links and sitemap coverage exist.
6. QualityThe page has a useful reason to be indexed.

Do not chase every crawler warning equally

A blocked money page, broken canonical or server error can stop a useful URL from performing. A tiny markup warning on an otherwise healthy page may be much less urgent. Prioritise by impact, affected URLs and business importance.

How to fix crawl errors and indexing issues in Search Console

  1. Open the Pages indexing report and select one affected URL from the error group.
  2. Inspect the exact preferred URL. Check crawl date, retrieval status, robots rules, indexing allowance and Google's selected canonical.
  3. Check the live response yourself. Does it return 200, 301, 404 or a server error? Compare the result with the version Google last crawled.
  4. Fix the actual cause: restore the page, correct the canonical, remove an accidental noindex, repair links, or resolve the redirect/server failure.
  5. Make sure the canonical page appears in the sitemap and has at least one relevant internal link.
  6. Test the live URL and request validation or recrawling when appropriate. Watch later reports rather than repeatedly submitting an unchanged broken URL.

Why does Google crawl but not index a page?

“Crawled - currently not indexed” does not automatically mean robots.txt is blocking it. Google may have crawled successfully but chosen not to index the page. Check originality, usefulness, canonical signals, near-duplicate pages and internal linking. Indexing is never guaranteed even when all technical checks pass.

Crawlability FAQs

Does submitting a sitemap fix crawl errors?

No. A sitemap helps discovery. It does not repair 404s, blocked pages, poor redirects or low-quality content.

Is noindex the same as a robots.txt block?

No. Robots rules restrict crawling; noindex tells a crawler not to index a page it can access. A robots block can prevent Google from seeing a noindex tag.

What should I fix first: crawl errors or keywords?

Fix status, crawl access, indexing and canonical problems first on important pages. Keyword and content improvements matter only after Google can reliably access the right document.

Final takeaway

Crawl and indexing problems become easier when you diagnose them in order. Confirm the preferred URL, check access and indexability, align canonicals, strengthen internal discovery, clean the sitemap and verify the result in Search Console. Only after those foundations are sound should you treat the issue as a content or authority problem.

NEED HELP WITH THIS?

Turn the checklist into a real plan for your website.

Share the site, the problem and the market. We can identify the highest-priority SEO or development work first.

Discuss your website →