UPDATE: I had always (wrongly) thought that Google wouldn't index pages that were blocked in robots.txt. But John Mueller clarified this for me (thank you John): robots.txt will control the crawling BUT NOT the indexing. Good explanation of that is here: ---------------- Google is now sometimes indexing pages DESPITE explicitly being blocked in robots.txt: Continue Reading ››