I don't focus on 'SEO' specifically anymore. Where I put my focus is on promoting my forums generally. It seems to me that Google is going to either like my site or not like it, and my focus should be on getting people to visit in the first place. Shawn makes a good point about making your site better, the other part of that is getting people to visit. Now, every site is going to be different in how you promote it. In a case like a forum about carpet cleaning, your target audience is very well defined and inviting people one at a time to visit might be releasable for you. If you were doing a forum about baseball, on the other hand, inviting people one at a time might not be as viable an option. Anyway, to get back to the point, focusing on SEO by itself isn't really a good strategy and it's not that you should ignore it just that it can't really be your main focus.
The only thing I've every really done is tweak the meta title and description contents to be keyword rich, but aside from that I don't there is much you really can do for forums (aside from getting your members to write keyword rich thread titles and that's about as easy as getting blood from a ....)
Like Shawn said, make sure the users are happy because they'll a) stay on the site longer and b) refer others via various social media channels, both of which are SEO signals Google takes into consideration.
We've found keeping the site as fast as possible is important too. Google takes that into account to some degree but more importantly the users like it, which leads right back to the first point in the previous paragraph.
Cache whatever data is frequently pulled or processed but infrequently changes
Set proper http cache control headers so the same resources are never needed to be requested again
Make sure your entire db fits in RAM and your my.cnf is properly tuned. Same for memcache, elastic search, etc
Be picky about what addons you install, make sure they can scale (or rewrite them so they do scale properly)
As for on page seo, xenforo does an excellent job out of the box. No complaints there
I don't do this anymore. Used to use vbulletin with vbseo in the days and that worked really well for me. Then I made a really big mistake and upgraded to vb 4 and stopped with vbseo because of the exploits in the product and their problems with continuing vbseo. Also I did not know I had rel=nofollow turned off (I got very ill during that period so I did not notice it at first). This gave me a massive penalty with google because of spambot users who registered on my site with links in their profiles. I went down from 100k unique vistors per month to 10k
Now I'm with Xenforo again and visitors are increasing already. But this might also have todo with that I switched from forum.domain.com to domain.com to avoid the penalty from google and I tried to get some good backlinks to help my site again. Pagerank still shows zero, so hopefully it will increase after next pagerank update.