No. of Recommendations: 4
This means that any crawler that scrapes for both search indexing and AI training will be turned away at the door, unless the site owner decides otherwise. I'm not entirely clear on whether they are doing this by being a hard front door guardian in front of the actual page content, or merely something like a manager of things like the robots.txt file which indicates to crawlers what's acceptable.
If the latter (unlikely), I presume the move would be entirely irrelevant.
Worth a read
Goofyhoofy writes on AI Tarpits"I have been inside rooms where decisions like this get made. The calculation is simple and it is ruthless. If the data is available and the legal risk is manageable, you take it. You do not ask. You scrape first and negotiate under legal pressure later, if at all. You assume the small operators won't have the resources to fight you, and you are usually right."If on the other hand they are truly blocking the content from crawlers successfully, that will certainly work. It certainly is interesting to see how the opposite of SEO can become an industry.
Jim
