If you're using Debian or Ubuntu I can help you very easily.
But before we get started do a ping test to your Dedicated Server or VPS Server. Make sure that its not a host issues that is making your site seem slow. IF things ping well and resolve well.... Let's move on....
First lets see what is using up so much resources.
apt-get install aptitude
aptitude install htop
top (make a screen copy from here)
htop (make a screen copy from here)
This will get me a basic understanding of what processes your using.
I also need to know the basic of your setup... ie.... Ram, cpu, apache version, php version, and mysql version. I would suggest 5.3.10 for php and mysql 5.5 or 5.6