.Google.com's John Mueller addressed a concern regarding why Google indexes pages that are prohibited coming from creeping by robots.txt and also why the it's risk-free to disregard the relevant Look Console reports regarding those crawls.Bot Visitor Traffic To Inquiry Guideline URLs.The individual talking to the question chronicled that crawlers were producing links to non-existent concern specification Links (? q= xyz) to webpages with noindex meta tags that are actually also blocked in robots.txt. What motivated the question is actually that Google.com is crawling the links to those pages, acquiring blocked through robots.txt (without noticing a noindex robots meta tag) then getting turned up in Google.com Look Console as "Indexed, though blocked out by robots.txt.".The individual asked the complying with question:." Yet here is actually the large question: why would Google.com index webpages when they can't also observe the content? What is actually the benefit in that?".Google's John Mueller validated that if they can not crawl the web page they can't observe the noindex meta tag. He also helps make an exciting reference of the internet site: hunt driver, urging to dismiss the end results because the "typical" consumers won't see those outcomes.He wrote:." Yes, you are actually proper: if our company can not crawl the web page, our team can't find the noindex. That claimed, if our team can't creep the webpages, then there is actually not a lot for us to mark. So while you might view a number of those webpages along with a targeted internet site:- question, the common user will not find all of them, so I would not fuss over it. Noindex is also great (without robots.txt disallow), it merely implies the URLs will certainly wind up being actually crept (as well as find yourself in the Look Console file for crawled/not catalogued-- neither of these conditions result in concerns to the rest of the internet site). The fundamental part is that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the limits being used the Internet site: search advanced hunt operator for analysis factors. Among those factors is actually since it is actually not connected to the normal search index, it is actually a distinct thing entirely.Google.com's John Mueller commented on the site hunt operator in 2021:." The quick answer is actually that a site: inquiry is actually not indicated to be total, nor utilized for diagnostics purposes.A web site query is actually a specific sort of hunt that restricts the end results to a particular internet site. It's generally only the word website, a bowel, and afterwards the internet site's domain name.This concern confines the outcomes to a details site. It is actually certainly not meant to be a detailed selection of all the webpages coming from that web site.".2. Noindex tag without using a robots.txt is actually alright for these type of situations where a bot is connecting to non-existent pages that are actually getting discovered by Googlebot.3. URLs with the noindex tag will definitely create a "crawled/not listed" entry in Browse Console and that those will not possess a bad effect on the remainder of the web site.Check out the question and also address on LinkedIn:.Why would certainly Google mark pages when they can't even see the material?Included Picture through Shutterstock/Krakenimages. com.