Nah, we run on less ram than that. If I turn memcached off, I can even get it right down around 1gb.Your forum is not small. You currently have almost 1 million posts and 270 users online. You probably need more memory.
You can install APC to speed up your forum. A memory cache will also help:
But I suspect you just need more memory.
It's not necessarily related, but with 1 million posts you can benefit from the Enhanced Search addon: