To prevent undesirable material in the lookup indexes, site owners can instruct spiders to not crawl specific information or directories with the standard robots.txt file in the root directory in the domain. Moreover, a site is usually explicitly excluded from the online search engine's database by utilizing a meta tag https://www.facebook.com/acourseinmiraclesdavid/