Robots.txt is a file that exists on the foundation directory of each Site and can be used to instruct search engines on which directories/documents of the website they will crawl and consist of inside their index. A sitemap is a simple file that lists all URLs on your internet site. https://news.rafeeg.ae