Only thing you can really do is changed "Job run trigger" to be "server-based" instead of "activity-based". The latter, which is the default, waits for a user to perform an action before checking for (and possibly executing) queued jobs. If you have a small/inactive site then it can take ages for these checks/executions to happen.
Otherwise as Tracy said this is pretty much how XF is designed. Imagine if all XF jobs and addon-provided jobs didn't use an enqueued jobs system, but instead were just various checks placed in the overall code. With a large enough site, and enough addons installed, doing simple things like just refreshing to board index could take ages, because the forum would be checking for user Trophies, check for user group Upgrades, rebuild user stats, etc when it doesn't really need to.