.Google's John Mueller addressed a concern about why Google.com indexes webpages that are actually prohibited coming from creeping by robots.txt and also why the it is actually safe to neglect the relevant Look Console files about those crawls.Crawler Website Traffic To Query Parameter URLs.The person talking to the concern recorded that robots were generating web links to non-existent question criterion URLs (? q= xyz) to webpages with noindex meta tags that are likewise shut out in robots.txt. What caused the question is actually that Google is crawling the web links to those webpages, getting shut out through robots.txt (without noticing a noindex robotics meta tag) at that point receiving shown up in Google Explore Console as "Indexed, though obstructed by robots.txt.".The individual talked to the adhering to question:." However right here is actually the significant concern: why would Google.com index webpages when they can't even view the information? What is actually the perk because?".Google.com's John Mueller verified that if they can't crawl the page they can not observe the noindex meta tag. He also creates a fascinating reference of the site: search operator, encouraging to overlook the outcomes given that the "typical" customers won't observe those results.He composed:." Yes, you're appropriate: if our company can't crawl the page, our experts can't see the noindex. That claimed, if our company can't creep the webpages, at that point there is actually not a great deal for us to mark. Therefore while you may view a few of those web pages with a targeted web site:- question, the average customer won't see all of them, so I would not bother it. Noindex is likewise fine (without robots.txt disallow), it merely means the URLs are going to find yourself being actually crawled (and also find yourself in the Search Console document for crawled/not recorded-- neither of these statuses create issues to the rest of the internet site). The integral part is actually that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the limits in using the Web site: search advanced hunt driver for diagnostic causes. Some of those explanations is actually because it is actually not linked to the frequent search index, it's a different point altogether.Google.com's John Mueller discussed the internet site hunt driver in 2021:." The brief solution is actually that a website: question is certainly not implied to become total, nor utilized for diagnostics purposes.An internet site inquiry is actually a certain kind of hunt that confines the results to a certain website. It is actually generally merely words website, a bowel, and afterwards the web site's domain.This query restricts the outcomes to a details internet site. It is actually certainly not implied to become a thorough compilation of all the pages coming from that site.".2. Noindex tag without using a robots.txt is actually great for these kinds of scenarios where a crawler is actually connecting to non-existent web pages that are obtaining found out through Googlebot.3. Links with the noindex tag are going to produce a "crawled/not recorded" item in Look Console and that those won't possess an unfavorable impact on the remainder of the website.Read the inquiry and respond to on LinkedIn:.Why would certainly Google index pages when they can not even view the information?Featured Photo by Shutterstock/Krakenimages. com.