On 2022 Apr 07 09:50:02, you wrote to me:

 BW> http://example.com/folder/robots.txt
 BW> "Not a valid robots.txt file. Crawlers don't check for robots.txt files in
 BW> subdirectories. "

this is true... robots.txt applies to the entire site so it is only valid when found in the root directory of the site... if it is located anywhere else, it is just another file...

)\/(ark

"The soul of a small kitten in the body of a mighty dragon. Look on my majesty, ye mighty, and despair! Or bring me catnip. Your choice. Oooh, a shiny thing!"
... People build walls to see who cares enough to break them down.
---