One of the reasons for using a subdomain is when you want that part of your site to be recognized (for example by search engines) as a site on itself. So we can assume that it has to be large, with many pages linking to each other within the subdomain. You could even do that for any large section of your site including the home page (http://frontpage.yourdomain.com) and redirect the main domain to it. Another assumption is that the content has to be different compared to the rest of the site.
To help with the decision you have to think about the consequences of actually having two (or more) domains for the same site, or just one domain with a folder structure. For example one consequence is that when you're link building, an incoming link to www.yourdomain.com does not directly affect forum.yourdomain.com. It will indirectly affect the forum subdomain when you link to it on your main domain. So perhaps not a great idea unless you have a really large site with a broad range of content that you want to split up and make it obvious in the domain.
We've been using a subdomain for the forums for many years, but I agree with Mike that it's not that obvious what the best way to go is. I may even decide to change it in the future to a folder structure.
.
