Take a look at Redmine. It's a rails app for project issue tracking and documentation that is free, open source, pretty extensible, and has a decently sized community posting how-tos, plugins (some free, some paid), some of which include a pretty nice helpdesk style ticketing system. There are also hosted solutions in case you didn't want to fiddle with installing it yourself, but I don't know off the top of my head of any that include the helpdesk type stuff off the bat.
The one caveat I'd offer is its UI is not as polished as something professionally done like XenForo or Zendesk (unless you do purchase one of the third party plugins for it that are nicely styled). But I've managed to scrape by on it at work for a few years now and it helps me keep a bit more sane with all the different balls that I need to keep juggling. I've got it set up with email issue creation/reply, so even seasoned email junkies who wouldn't dare touch a web app for commenting on a ticket can join in the fun
Not sure of the possibility of integrating it with XenForo, as I've never had an occasion to use both of them in the same domain, but I'll bet there's a way to integrate the authentication system if that is desirable.