On 2022 Apr 05 16:24:54, you wrote to me:

 BW>>> To my surprise, they seem to try to enter a /robots.txt subdirectory

 ml>> that's how they decide if it is a file or a directory... if it is a
 ml>> file, then they download it...

 BW> Yes, I suspected that might be the case... I wonder, though, if they
 BW> do anything special if the cd succeeds

if the cd succeeds they do an ls to see if there's file and directories in it... if there are, they download them and continue walking through the directories...

 BW> or if they just conclude "not a file" and skip it. (According to the
 BW> spec, they shouldn't look for anything inside that directory, if it
 BW> exists.)

what spec is that?

)\/(ark

"The soul of a small kitten in the body of a mighty dragon. Look on my majesty, ye mighty, and despair! Or bring me catnip. Your choice. Oooh, a shiny thing!"
... If all you have is a hammer, everything looks like a nail.
---