Robots meta tag, data-nosnippet & X-Robots-Tag specifications: Documentation
The robots meta tag, the data-nosnippet HTML attribute, and the X-Robots-Tag HTTP header are tools webmasters can use to manage how their content appears in Google Search results. These mechanisms instruct Google’s crawlers on how to index pages, display snippets, or serve cached versions.
While Google respects these directives for indexing and snippet display, it’s important to understand they do not prevent content from being crawled. For complete content exclusion, consider robots.txt or requiring user authentication.
Using the robots meta tag
To use the robots meta tag, add it inside the <head> section of your HTML document:
htmlCopyEdit<head>
<meta name="robots" content="noindex, nofollow">
</head>
This example instructs crawlers not to index the page or follow links. You can specify different directives:
htmlCopyEdit<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">
<meta name="robots" content="noarchive">
<meta name="robots" content="nosnippet">
<meta name="robots" content="max-snippet:50">
<meta name="robots" content="max-image-preview:standard">
<meta name="robots" content="max-video-preview:0">
Multiple meta tags on the same page
If multiple robots meta tags are present, only the first one in the source code will be recognized. It is best to use only one robots meta tag per page to avoid confusion.
htmlCopyEdit<!-- Only the first one is considered -->
<meta name="robots" content="noindex">
<meta name="robots" content="index, follow">
Using the X-Robots-Tag HTTP header
You can apply indexing directives through HTTP headers as well, which is especially useful for non-HTML files like PDFs or images:
httpCopyEditX-Robots-Tag: noindex, nofollow
For example, in Apache’s .htaccess:
apacheCopyEdit<Files "private.pdf">
Header set X-Robots-Tag "noindex, noarchive, nosnippet"
</Files>
Or in nginx:
nginxCopyEditlocation = /private.pdf {
add_header X-Robots-Tag "noindex, noarchive, nosnippet";
}
data-nosnippet HTML attribute
You can use data-nosnippet to prevent a specific part of your page from appearing in a search snippet:
htmlCopyEdit<span data-nosnippet>This text will not appear in the search snippet.</span>
Use this to avoid leaking sensitive or unnecessary content without removing the whole page from the snippet.
Understanding and applying rules
Here’s a table to understand the behavior of different tags and headers.
| Directive | Description |
|---|---|
noindex | Prevents the page from being indexed. |
nofollow | Stops Googlebot from following links on the page. |
nosnippet | Prevents snippet display and stops caching of the page. |
noarchive | Stops the cached version from being displayed in search. |
max-snippet:[number] | Limits the number of characters in the snippet. |
max-image-preview:[...] | Controls image preview size: none, standard, or large. |
max-video-preview:[...] | Limits video preview duration in seconds. |
Example: Limit snippet to 50 characters
htmlCopyEdit<meta name="robots" content="max-snippet:50">
Example: Limit image preview size
htmlCopyEdit<meta name="robots" content="max-image-preview:standard">
Example: Prevent snippet and archive but allow indexing
htmlCopyEdit<meta name="robots" content="nosnippet, noarchive">
Practical implementations of X-Robots-Tag
To apply the X-Robots-Tag via server configuration:
Apache
apacheCopyEdit<FilesMatch "\.(pdf|doc)$">
Header set X-Robots-Tag "noindex, noarchive, nosnippet"
</FilesMatch>
nginx
nginxCopyEditlocation ~* \.(pdf|doc)$ {
add_header X-Robots-Tag "noindex, noarchive, nosnippet";
}
Combining robots meta with scripting and markup rules
These directives can be combined with other content management or scripting strategies. For example:
htmlCopyEdit<div data-nosnippet>
Confidential pricing data goes here.
</div>
Or combining meta with data-nosnippet:
htmlCopyEdit<head>
<meta name="robots" content="index, nosnippet">
</head>
<body>
<p data-nosnippet>This section won't be shown in snippets.</p>
</body>
These tools give site owners detailed control over how content is discovered, displayed, and cached in Google Search.
Leave a Reply