The times would have to correspond with when the cron runs, which is 5:37 AM GMT, so you'll need to look around that vicinity.
If rebuilding manually works fine, that makes troubleshooting decidedly more difficult and it might point to environment variations as the cause. You can manually trigger the cron and then wait a little while to see if it finishes. You'll want to start using the sitemap options to remove all the content types and then add them back one by one to see when it stops building successfully. The last entry enabled is likely the cause.
If it's getting stopped half way through, that means that PHP is being killed before execution completes (and generally without being given the ability to clean itself up). Running through the command line is unlikely to trigger the same scenario if it is indeed a process watcher that's killing it. You can trigger the cron within XF though and then watch the log to see it being built.
I have a suspicion that there may be environmental effects here, so the surrounding details may be relevant (what else is happening on the server at the time, load average, etc). It may be worth seeing what happens around the time the build fails.
There should be a log on the server somewhere to give you an idea what's happening. If it's sporadically working, that does seem to point to something on the server, rather than the code (though it's possible).
If you haven't already, you will likely need to start debugging by building the content types included up one by one over a period of time to see if there's a particular type that is causing it to fail.