Technical SEO
Google indexing pages despite explicitly blocked in robots.txt
UPDATE: I had always (wrongly) thought that Google wouldn’t index pages that were blocked in robots.txt. But John Mueller clarified this for me (thank you John): robots.txt will control the crawling BUT NOT the indexing. Good explanation of that is here: —————- Google is now sometimes indexing pages DESPITE explicitly Read more