Google updated its Googlebot documentation to clarify file size limits for crawling and indexing. The Googlebot page now states a 2 MB limit for HTML and supported text-based files, down from the previously documented 15 MB limit. Will this documentation update affect your website SEO? The answer is no, as a typical HTML webpage is well under 2 MB when uncompressed. Let’s dive deeper into the topic.
What Is Crawling and Why Is It Important for Your Website?
Crawling is how Google discovers and reads your website content. It usually starts by fetching the page’s HTML, then requesting other files such as CSS and JavaScript to understand how the page is built and displayed.
Google can only process and index content that it is able to crawl properly, which means crawling is an important first step in helping your page appear on Google Search.
Now that we have covered the basics of crawling, let’s get into the main topic.
What Changed in Google’s Crawling Documentation?
Google reorganised its crawler documentation and updated the file size limits across different pages. The documentation now lists a 2 MB limit for HTML and supported text-based files, with a separate 64 MB limit for PDFs.
Google also states that these limits are based on the uncompressed file size.
What Does Uncompressed File Size Mean?
Uncompressed file size is the full size of a file once it has been unpacked, not the smaller compressed version sent from your website’s server to Google.
In other words, even if a file is reduced in size when it is transmitted, Google applies the crawl limit based on the file’s full size after it is unpacked.
Major Crawler Update or Just A Documentation Update?
Google clarified that this is not a new crawling restriction for Googlebot. The 15 MB default limit still exists within Google’s broader crawling infrastructure. Read Google’s official documentation on the different types of crawlers it deploys.
This means most websites are not facing a new technical constraint. Instead, Google is making its documentation clearer so site owners understand how different crawlers operate.
To avoid confusion, focus on Googlebot, which is the most important crawler for anything search-related.
Understanding the 2MB Limit Per File
From what I saw on LinkedIn, many marketers seemed to misunderstand Google’s updated documentation and assumed that an entire webpage has a total 2 MB crawl limit. That is not what Google means.
Instead, the 2MB limit applies to each individual supported file Googlebot fetches. In simple terms, Google does not add together your HTML, CSS, JavaScript, and image files into one shared 2MB allowance for the whole page.
For example, your page may contain:
- one HTML file
- one CSS file
- one JavaScript file
- several image files
Googlebot handles these resources separately. This means the HTML file has its own limit, while CSS and JavaScript files are fetched separately as needed.
This is an important distinction because many website owners may wrongly assume that having a large page with many assets automatically means Google cannot crawl it properly.
How to Check If Your Pages Exceed the Limit
You can audit your site’s file sizes using crawling tools like Screaming Frog (SF):
- Look at the Size column, which shows the uncompressed file size. This is the number that matters for the 2 MB indexing limit.
- The Transferred column shows the compressed size during transmission, which is different from what Google processes.
- If you have JavaScript crawling enabled in Screaming Frog, you will also see Total Transferred. This includes all the files loaded for rendering the page. However, the 2 MB limit does not apply to the total page weight. It applies per supported file type.
For most sites, uncompressed HTML files will remain well below 2 MB. It is possible that some JavaScript files or images may exceed 2 MB, but oversized assets should be optimised regardless of crawling limits, as they slow down page performance.
How Does This Affect Your Website?
Most websites will not be affected by this limit. To put the limit into perspective, typical HTML files are far smaller than 2 MB.
Only unusually large or heavily scripted pages are likely to approach the limit. Here are some examples:
- Websites with heavy JavaScript frameworks
- Single-page applications (React, Angular, Vue) with large inline scripts
- Page builders that inject excessive code before content
- Pages loading large JSON data server-side
- Infinite scroll pages that render too much content upfront
What This Means for Business Owners?
The key to ensure your web content are crawled by Google is to ensure that no single file exceeds the threshold for its file type.
Use crawling tools like SF to identify any page approaching or exceeding the 2MB uncompressed file limit.
If you are still unsure how to check whether your page files are too large, speak to a web developer or a technically trained SEO for advice.
How to Check If Important Content is Being Indexed?
If you are unsure whether important content is being indexed, test it directly. Search for a unique sentence located near the end of the page. Type the following into Google Search: site:yourdomain.com “your exact sentence here”
If it appears in search results, Google has processed that portion of the file. This approach provides clearer validation than relying on file size alone.
