On 2022 Apr 07 09:50:02, you wrote to me:
BW> http://example.com/folder/robots.txt
BW> "Not a valid robots.txt file. Crawlers don't check for robots.txt files in
BW> subdirectories. "
this is true... robots.txt applies to the entire site so it is only valid when found in the root directory of the site... if it is located anywhere else, it is just another file...
)\/(ark
"The soul of a small kitten in the body of a mighty dragon. Look on my majesty, ye mighty, and despair! Or bring me catnip. Your choice. Oooh, a shiny thing!"
... People build walls to see who cares enough to break them down.
---
* Origin: (1:3634/12.73)
|