Varnish is more suitable for an application that is slow. XenForo is not slow, so you basically are just adding an additional layer of complexity and something that can break for no real upside/gain.
Varnish is more for when you can't fix the slowness in your application.
Also - last time I looked, Varnish didn't work with HTTPS (maybe fixed now, but still not something I would use).