Hi! For those of you that come across this in the future and want to save a few hours, here's a working config I figured out for FrankenPHP.
The config should be fairly similar for plain Caddy. This makes Friendly URLs work as well.
handle /xf/* {
# Internal only
@forums_internal path...