Technical SEO is the work of making a website easy for search engines to crawl, render and index. Google has no central registry of web pages; it discovers URLs through links from pages it already knows and through sitemaps submitted by site owners.[1]
Crawling
Google's crawler renders pages and runs JavaScript in a recent version of Chrome.[1] Its crawl budget, the set of URLs Google can and wants to crawl on a site, depends on how much load the server can handle and on demand driven by the site's size, freshness and popularity; Google says crawl budget is mainly a concern for very large or fast-changing sites.[6] Pages that return 4xx status codes are not indexed, and persistent server (5xx) errors slow crawling and can eventually cause pages to be dropped. A "soft 404" is a page that returns a success code but whose content looks like an error.[7]
robots.txt and noindex
A robots.txt file tells crawlers which parts of a site they may crawl, mainly to keep servers from being overloaded. Google states that it is not a mechanism for keeping a page out of search results: a blocked URL can still be indexed if other pages link to it.[8] The Robots Exclusion Protocol was standardized by the IETF as RFC 9309 in September 2022, which notes that its rules are not a form of access authorization.[2] Google supports only the user-agent, allow, disallow and sitemap fields, ignores crawl-delay, and reads files up to 500 KiB.[9]
To keep a page out of Google, site owners use a noindex rule in a robots meta tag or an X-Robots-Tag HTTP header, which also works for files such as PDFs. The page must not be blocked by robots.txt, or Googlebot will never see the rule; Google does not support noindex inside robots.txt.[10]
Sitemaps
An XML sitemap lists a site's URLs for search engines. Under the sitemaps.org protocol each file may contain up to 50,000 URLs and 50 MB uncompressed,[3] limits Google applies to all sitemap formats. Google ignores the priority and changefreq fields and uses lastmod only when it is consistently accurate.[11]
Canonical URLs and redirects
When several URLs show the same or very similar content, Google chooses one as canonical. Redirects and rel="canonical" links are strong signals of the preferred URL and sitemap inclusion a weak one, but all are treated as hints rather than commands.[12] Permanent redirects (301 and 308) tell Google the target should be canonical, while temporary redirects (302 and 307) let the original URL stay in results; server-side redirects are the most reliable.[13]
HTTPS and mobile-first indexing
In August 2014 Google announced that it would use HTTPS as a "very lightweight" ranking signal, initially affecting fewer than 1% of queries.[14] With mobile-first indexing, Google uses the mobile version of a page, crawled by its smartphone crawler, for indexing and ranking; Google announced in October 2023 that the switch to mobile-first indexing was complete.[15]
Page experience and Core Web Vitals
Core Web Vitals are three metrics of real-world user experience, measured at the 75th percentile of page loads:[4]
| Metric | Measures | "Good" threshold |
|---|---|---|
| Largest Contentful Paint (LCP) | Loading | 2.5 seconds or less |
| Interaction to Next Paint (INP) | Responsiveness | 200 milliseconds or less |
| Cumulative Layout Shift (CLS) | Visual stability | 0.1 or less |
INP replaced First Input Delay as a Core Web Vital on March 12, 2024.[16] Google says its ranking systems use Core Web Vitals but that good scores do not guarantee high rankings and there is no single page-experience signal.[17]
JavaScript and international sites
Google processes JavaScript pages in three phases, crawling, rendering and indexing, and still recommends server-side rendering or pre-rendering where possible, along with real links and meaningful HTTP status codes.[18] For sites in several languages or regions, hreflang annotations tell Google which pages are localized versions of each other; each version must link back to the others or the annotations may be ignored.[19]
Google Search Console
Google Search Console shows site owners how Google crawls, indexes and serves their sites, with reports on indexing, search performance, rich results and Core Web Vitals, a URL inspection tool and notices of manual actions.[5] Its page indexing report explains why pages are not indexed, such as robots.txt blocks, noindex rules, 404 errors or duplication, and notes that a page not being indexed is not always a problem.[20]
References
- ^ ^ ^ "In-depth guide to how Google Search works". Google Search Central, updated December 18, 2025. developers.google.com
- ^ ^ Koster, M.; Illyes, G.; Zeller, H.; Sassman, L. (September 2022). "Robots Exclusion Protocol" (RFC 9309). IETF / RFC Editor. rfc-editor.org
- ^ ^ "Sitemaps XML format". sitemaps.org. sitemaps.org
- ^ ^ Walton, Philip (updated October 31, 2024). "Web Vitals". web.dev. web.dev
- ^ ^ "Get started with Search Console". Google Search Central, updated December 10, 2025. developers.google.com
- ^ "Optimize your crawl budget". Google Search Central, updated July 22, 2026. developers.google.com
- ^ "How HTTP status codes affect Google's crawlers". Google Search Central, updated February 4, 2026. developers.google.com
- ^ "Robots.txt introduction and guide". Google Search Central, updated December 10, 2025. developers.google.com
- ^ "How Google interprets the robots.txt specification". Google Search Central, updated October 23, 2026. developers.google.com
- ^ "Block search indexing with noindex". Google Search Central, updated December 10, 2025. developers.google.com
- ^ "Build and submit a sitemap". Google Search Central, updated July 8, 2026. developers.google.com
- ^ "How to specify a canonical with rel="canonical" and other methods". Google Search Central, updated July 10, 2026. developers.google.com
- ^ "Redirects and Google Search". Google Search Central. developers.google.com
- ^ Bahajji, Zineb Ait; Illyes, Gary (August 7, 2014). "HTTPS as a ranking signal". Google Search Central Blog. developers.google.com
- ^ Mueller, John; Kalush, Nir (October 31, 2023). "Mobile-first indexing has landed". Google Search Central Blog. developers.google.com
- ^ Wagner, Jeremy; Viscomi, Rick (January 31, 2024). "Interaction to Next Paint becomes a Core Web Vital on March 12". web.dev. web.dev
- ^ "Understanding page experience in Google Search results". Google Search Central, updated September 22, 2026. developers.google.com
- ^ "Understand the JavaScript SEO basics". Google Search Central, updated March 4, 2026. developers.google.com
- ^ "Tell Google about localized versions of your page". Google Search Central, updated September 21, 2026. developers.google.com
- ^ "Page indexing report". Search Console Help. support.google.com