InnoDB works best when you can give it at least as much RAM as you have data in the database - essentially it caches everything in RAM.
If your tables are using InnoDB, and you have, say, 2.6GB of data in your InnoDB tables, then assigning something like 3GB RAM to InnoDB caching will really improve performance.
That is of course assuming you have sufficient RAM available on the machine - if you allocate too much, you'll cause instability on the machine - I accidentally allocated too much to the InnoDB cache last week and the Linux OOM (Out Of Memory) manager started killing the MySQL process occasionally to manage it. I dropped the allocation back a bit (and moved one of my largest databases to its own server) and it's been stable ever since.
Have you been running
http://mysqltuner.com/ ?
It's worth going back to default settings, letting it run for 24 hours and then running mysqltuner, changing the settings it suggests, restart the server and let it run for another 24 hours or so before repeating the process until you either run out of RAM to allocate, or it has no more suggestions.
Also, don't over-allocate your key buffer or join buffer - these are per-connection settings and so you'll end up using a heap of RAM for not much performance benefit if you allocate too much, especially if you have a lot of connections allocated. If connection usage is fairly constant like mine is, I generally try and allocate sufficient connections so that the max connections used is never more than 80% of what I allocated. Too much more and you're just wasting RAM that is allocated for those unused connections.
Most importantly - check out how much IO activity there is on your swap drive. Having swap allocated isn't a problem - it's how much it gets used which will really slow your machine down. If your machine is constantly swapping, it probably means you've over-allocated memory (or just need more of it), and your disk IO is suffering because the machine is spending too much time doing IO to the swap drive. This is especially bad if you aren't using SSDs.
I have a separate database server serving 25 websites running on an 8GB Linode (4 CPU cores, SSD drives), it has 4.6GB of data in InnoDB tables, so I've got 5GB allocated to the InnoDB buffer pool.
Most of the sites are low traffic, but I have a couple of XenForo sites getting 20K page views per day and one XenForo site getting 40K page views per day - all being served from the same database server.
