The key thing to understand is what minimum permissions you need to set in order to accomplish what you are trying to do. Sometimes this isn’t obvious, but the documentation may give you some clues. Googling for advice might help, too.
For a CGI script, you want to be able to read, write, and execute the CGI file yourself. For other users, you want to allow only read and execute operations. When you OR the bitfields (execute=1, write=2, read=4) together for user, group, others, that gives you 755.
For static HTML pages, you need to read and write the files, but others only need to read them. No one needs to execute them. Thus, you use 644.
For directories, you need to set the execute bit if you want to allow users to list the contents of the directory. Remember that users aren’t just interactive users. Programs like HTTP servers execute operations as a user on your files and directories on behalf of people browsing your website.
Normally, you set the permissions for group and others to be the same. If your server has a group of users to which you want to give higher privileges, then you would obviously need to use different values for group and others. You can use chgrp to change the group ownership for a file or directory. Type ls-l in a Unix shell to see the current owner/user and group for a file or directory. Type groups to see what group(s) you are in.
In almost every case, you want to set the privileges to the absolute minimum level that will accomplish your objective. For CGI scripts, that means using 755 instead of 777.
With respect to your original question, 766 makes the most sense to me for a directory. That means you can read, write, and execute (i.e., list the directory contents) and everyone else can read and write. I would go with 766, although I can’t say this from first hand YaBB experience. If a user can’t access a directory, you wouldn’t be getting a 500 error. Some CGI scripts require 766 or 744 for data files that they write.
Hopeful this was helpful and not overly pedantic,