Google's New Video Explains What robots.txt Files Actually Support
· Digital Living
Google's Gary Illyes and Chery Prommawin published the second video in their Non-Canonical Cabinet series, covering what robots.txt supports and what it does not. The key point is that Google's support for a directive does not mean other search engines or AI crawlers will honor it.
The video is titled What can you actually put in a robots.txt? It walks through which features and directives robots.txt files support and which they do not. It is the second episode in the Non-Canonical Cabinet series.
The coverage reportedly touches on topics such as AI crawlers, emojis and international URLs, among other things. The source gives no detail on what is said about each, so the video itself is the place to check specifics.
The main caution is that Google may support a given feature or directive while other search engines or AI crawlers do not. A rule that works for Google is not guaranteed to be respected elsewhere.
For businesses, this matters because robots.txt is often treated as a universal switch for controlling crawler access. The video suggests it is better to treat it as a set of instructions that each crawler may interpret differently.
Takeaway for businesses: Review your robots.txt file against what is actually supported, and do not assume that rules Google follows will be followed by other search engines or AI crawlers.
Sources: https://www.seroundtable.com/google-video-robots-txt-rules-42355.html