Crawling December: HTTP Caching Insights & Recommendations
As the internet continues to grow, so do the challenges of web crawling. While Google’s crawling infrastructure has always supported heuristic caching, the percentage of cacheable requests has dropped over the years. Ten years ago, only 0.026% of total fetches were cacheable—a modest figure—but today, that number is even lower at 0.017%. This trend highlights the need for better caching practices.
Why is Caching Important?
Caching is a vital component of web performance, enabling:
- Faster page loads: Cached content allows users and crawlers to retrieve data quickly on revisits.
- Reduced resource consumption: Caching minimizes server load, saving computation power and bandwidth.
- Environmental benefits: Lower resource usage translates to less energy consumption.
For websites with rarely changing content, enabling caching can significantly improve crawling efficiency. Google’s crawlers rely on HTTP caching standards, including:
- ETag and If-None-Match headers.
- Last-Modified and If-Modified-Since headers.
To maximize efficiency, it’s recommended to use both mechanisms, but ETag is often preferred due to its flexibility and reduced error potential.
ETag and If-None-Match
The ETag (Entity Tag) header is a unique identifier for a specific version of a resource, typically generated as a hash or string. It signals to crawlers whether content has changed.
How it works:
- When a crawler first accesses a URL, the server includes an ETag value in the response.
- On subsequent requests, the crawler sends the If-None-Match header with the ETag value from the previous response.
- If the server finds the ETag unchanged, it responds with HTTP 304 (Not Modified) and skips sending the content body.
Benefits:
- Reduces server load by skipping content generation.
- Saves bandwidth by avoiding unnecessary data transfer.
- Improves response time for users and crawlers.
Last-Modified and If-Modified-Since
The Last-Modified header indicates the timestamp of the last content update. It works similarly to ETag but uses date-based validation.
How it works:
- The server includes the Last-Modified timestamp in the response.
- On subsequent requests, the crawler sends the If-Modified-Since header with the saved timestamp.
- If the content hasn’t changed, the server responds with HTTP 304 (Not Modified).
Recommendations for Last-Modified:
- Use the standard HTTP date format (e.g.,
"Fri, 4 Sep 1998 19:15:56 GMT"). - Set the
max-agedirective in the Cache-Control header to specify how long content is considered fresh (e.g.,Cache-Control: max-age=94043).
Examples
To better understand how these caching mechanisms work, let’s compare them side by side:
| ETag/If-None-Match Workflow | Last-Modified/If-Modified-Since Workflow |
|---|---|
| Server’s Initial Response: | Server’s Initial Response: |
HTTP/1.1 200 OK | HTTP/1.1 200 OK |
ETag: "34aa387-d-1568eb00" | Last-Modified: Fri, 4 Sep 1998 19:15:56 GMT |
| Crawler’s Conditional Request: | Crawler’s Conditional Request: |
GET /example | GET /example |
If-None-Match: "34aa387-d-1568eb00" | If-Modified-Since: Fri, 4 Sep 1998 19:15:56 GMT |
| Server’s Conditional Response: | Server’s Conditional Response: |
HTTP/1.1 304 Not Modified | HTTP/1.1 304 Not Modified |
Both mechanisms allow the server to validate cached content and avoid sending unnecessary data, leading to improved efficiency.
Why You Should Enable Caching
Enabling HTTP caching is a win-win for website owners and users:
- For website owners: It reduces hosting costs and server resource usage.
- For users: It ensures faster page loads and better experiences.
To implement caching, consult your hosting provider or CMS documentation for setup instructions. By enabling caching, you’ll make your website more efficient, environmentally friendly, and crawler-friendly.
For detailed documentation and community discussions on caching, refer to Google’s official documentation.
Leave a Reply