Well regardless of how you trigger the cron, it runs the same way. However, I find it strange that doing it via rebuild caches is fine as it's fundamentally the same approach.
You may need to temporarily rename deferred.php (so only you access it) and manually trigger the sitemap cron. Now access now-renamed-deferred.php manually in your browser. You should see an indication when it runs. Keep refreshing that until you get an error or unexpected output or the sitemap is built in the ACP. If it errored, what happened? After you do this, you need to rename the file back so that crons will be triggered automatically as necessary.