How to Create file, What is , What are the main points to Create

A file tells search engine crawlers what URLs the crawler can access on your site. This is mainly used to avoid overloading your site with requests; This is not a system to keep any web page out of Google. To keep a web page away from Google, block indexing with Noindex or password protect the page. A file can be found at the base of your website. For example, for the website the file is located at The file is a plain text file that follows the robots exclusion standard. The file contains one or more rules. Each rule restricts or allows access to a single file location on the domain or subdomain where the file is hosted. A) Here is a simple file with two rules: Useragent: Googlebot Disallow:, nogooglebot, Useragent: Allow:, Sitemap: B) Basic guidelines for creating a file There are four