I thought ridey.net had a fairly extensive robots.txt file, then I came across this one on the White House website. Surely the longest on the web at 1,675 lines! Got something to hide, George buddy?
I thought ridey.net had a fairly extensive robots.txt file, then I came across this one on the White House website. Surely the longest on the web at 1,675 lines! Got something to hide, George buddy?