~kevin~ Member Dec 3, 2019 #1 Should the sitemap location in robots.txt have a $ at the end? Example - Sitemap: url/sitemap.php$ I was reading through some sitemap suggestions and one was to use a dollar sign to denote the end of a url. Does anyone do this? Have ya'll heard of it? Thanks in advance.
Should the sitemap location in robots.txt have a $ at the end? Example - Sitemap: url/sitemap.php$ I was reading through some sitemap suggestions and one was to use a dollar sign to denote the end of a url. Does anyone do this? Have ya'll heard of it? Thanks in advance.
Andrej Well-known member Dec 3, 2019 #2 What does the dollar sign mean in robots.txt I am curious about a website and want to do some web crawling at the /s path. Its robots.txt: User-Agent: * Allow: /$ Allow: /debug/ Allow: /qa/ Allow: /wiki/ Allow: /cgi-bin/loginpage Disallow: /... stackoverflow.com Upvote 0 Downvote
What does the dollar sign mean in robots.txt I am curious about a website and want to do some web crawling at the /s path. Its robots.txt: User-Agent: * Allow: /$ Allow: /debug/ Allow: /qa/ Allow: /wiki/ Allow: /cgi-bin/loginpage Disallow: /... stackoverflow.com