It is likely that no one has answered you because they are not using Group-Office . From their website, I see that the software is available in a commercial and a community edition; which version are you using? Since no one has answered, I'll try to help.
From a quick look at the documentation on the site, and having been around DreamHost for quite a while, I'm willing to take a shot at answering some of your questions
[color=#CC0000]Disclaimer[/color]: I have not installed this software on Dreamhost, and should you choose to rely on any of my answers, you do so oat your own risk. While it is likely that the worst that could happen as a result of attempting to configure this software with the information I provide is that it won't work, it is possible that this software could crash your computer, kill your dog, rape your cat, and cause you all kinds of grief - if it does anything "bad" at all, I won't be responsible. My advice is worth exactly what you have paid for it! .
Those settings look generally ok with one major exception. Please, for your own sake and for the safety of others sharing your server, DO NOT run this (or, for that matter, ANY PHP software with Register_Globals enabled. Since you have a custom PHP install, it should be trivial for you to remedy this; Group-Office should run fine with Register_Globals "off. If it doesn't, please don't run it on a Dreamhost shared server.
Here is what I recommend trying for these settings:
This is fairly standard for many PHP applications that send mail. What the instructions appear to be trying to tell you (the English is pretty bad!) is that you can configure the system several different ways, depending upon what you set the "Mailer" to be.
You did not indicate whether you have complied your custom PHP with IMAP extension or not, but note that you will need to do so to use IMAP (recommended by the Group-Office docs). In light of that, I'll only address the simplest setup here.
If you set the mailer choice to be PHP(Mail) Function, you DO NOT need to configure the following SMTP settings (SMTP server, SMTP port, SMTP username, and SMTP password). By using this choice, you won't be able use CC and BCC headers in mail handled by the system). To use this set-up, which is the simplest, just choose the PHP(Mail) Function for the "Mailer" choice. Note that you need to set a maximum size for attachments (it must be less than the value specified for what "Current PHP configuration allows" shows. Once these things are done, you are done configuring this section.
I can't answer all these for you (I don't know if you have IMAP available in your PHP5 installation), but you likely will be able to get it to work as follows:
Name: The Name you want to display for the email user
E-mail: Your email address (assuming firstname.lastname@example.org - your DreamHost account where you are running this software)
Type: (choices are IMAP / POP3) - POP3 should work, though that and IMAP depend upon your PHP compilation)
(Select Box) SSL - unless you have an dedicated IP and a certificate installed on DH do not check this
(Select Box) Don't validate certificate - only relevant if the SSL box is selected
Port: 110 (for POP3)
Root mailbox: this is likely your default Dreamhost email user account you plan to use for this system - email@example.com
Username: that root email account's user name
Password: and it's password
Signature: Optional signature to be appended to email messages
(Select Box)Automatically check this account for new mail. - that is just a matter of preference.
A couple of thoughts here:
Depending upon how many users you plan to support on this system, IMAP could be problematic in terms of inbox size and disk storage accumulation for email. Frankly, for this type of a system I would recommend POP3, but YMMV. By jusdicious and managed use of the "Leave mail on server" settings in Thunderbird and MS*FT products, people can still effectively use the remote email from anywhere - they just have to plan a little differently.
You might also want to take a careful read through the documentation at the provider's website - there are 8weveral* configuration issues that should be checked (Here, for example).
I don't know if any of this of useful at all, but I wanted to try to give what help I could. IF you are using the "Professional" version of the software, support should be available from the vendor. I also think you are likely to get valuable help from the vendor's community forum. DH is a "large" host, it is likely that someone in those forums has 'already done it here" or, if not, has done it on other shared hosts and, being familiar with Group-Office, can be of better help than I.