Technical SEO files: robots.txt, sitemaps, canonicals and hreflang

robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it, just without a description. To keep a page out of the index, let it be crawled and add a noindex robots meta tag or X-Robots-Tag header instead.

Each crawler obeys only the most specific User-agent group that matches it. Within a group, Google applies the most specific (longest) matching rule, and when an Allow and a Disallow rule are equally specific, Allow wins. Paths are case-sensitive.

An XML sitemap lists the canonical URLs you want search engines to find, with an optional lastmod date. One file may hold up to 50,000 URLs or 50 MB uncompressed; larger sites split the list across several files tied together by a sitemap index.

List only URLs that return 200, are their own canonical and are not noindexed. A sitemap that contradicts a page's own signals sends mixed messages and wastes crawling. Reference the sitemap from robots.txt and submit it in Search Console.

When the same content is reachable at several URLs (tracking parameters, http and https, with and without a trailing slash), rel="canonical" tells search engines which one to show. It is a strong hint rather than a directive: if redirects, internal links or the sitemap point somewhere else, Google may choose a different URL.

Use one absolute canonical URL per page, and point it at a page that returns 200 and is indexable. A canonical pointing at a redirect, a 404 or a noindexed page is ignored.

hreflang annotations connect translated or regional versions of a page so searchers get the right one. Every version lists all the versions, itself included, and the links must be reciprocal: if page A names page B, page B must name page A.

Codes are an ISO 639-1 language, optionally followed by an ISO 3166-1 region: en-GB is valid, en-UK is not. x-default marks the page for everyone who matches no listed version.

The title tag and meta description are what searchers usually see in results, though Google may rewrite either when it doesn't describe the page or match the query. Keep titles specific and roughly 50–60 characters so they aren't cut off.

Open Graph and Twitter (X) card tags control the preview shown when a link is shared. A 1200×630 pixel image works across most networks; always set og:title, og:description, og:url and og:image.

JSON-LD describes a page's entities, such as an article, a product, an organization or a breadcrumb trail, in a script tag that search engines read. The markup must describe what is visible on the page.

Valid markup makes a page eligible for rich results; it does not guarantee them, and Google limits some types (FAQ rich results, for example, now appear only for well-known government and health sites).

Use a permanent redirect (301 or 308) when a URL has moved for good, and a temporary one (302 or 307) only when the original will come back. Point old URLs straight at the final destination: every extra hop adds latency, and Google stops following a chain after about ten hops.