I hit the same disappearing behavior, and didn’t find a way around it. Once you access the directory and authorize, then the listing includes the directory again. Not what I wanted, but seems to be the way it works.
For the 404 error, should be a bug in your htaccess, try looking at your error log after you access it. (in logs/domainname/http/error.log)