I'm not familiar with xen porta and whether it changes the route controller at all, but in a default installation, you would have to install XenForo in /forum to achieve mydomain/forum.
Be aware though that the forums will have a URL such as mydomain/forum/forums/name-of-forum.