I've been having the same issue recently; I have at minimum an extra 250mb of RAM over what I use during peak periods, yet my VPS has rebooted several times in the last 24 hours, forcing me to put the RAM limit up.
However, I don't believe I should need it to be as high as it is; has something changed recently that could have made the stability robot more sensitive to spikes in usage? When I run top on my server my own processes (i.e - those under my username) are using less than a quarter of my RAM allocation, with users such as root, postfix and daemon bringing me to about a third. The rest is all dhapache, a big chunk of which I believe is actually disk caching which shouldn't simply decrease as RAM usage for other VPS users increases. It seems to me that the stability robot may be including disk cache in its checks, either that or the disk cache is failing miserably at backing off when other processes require more RAM.
Another possibility is that I've been having a lot of trouble with ImageMagick creating huge files in /tmp, do the contents of /tmp (which are stored in RAM) count against a VPS' RAM usage? Even so, I should have more than enough free RAM to cover /tmp suddenly filling up, since it is limited to 128mb.