The precise "ideal" values are very dependent on your server setup. It depends what user runs the scripts, the level of access you have to the server, and the general comfort level you might have dealing with chmod/chown as root.
The only thing that XF really cares about is whether the user running PHP can write to the files. If you don't set a preference in config.php, XF will make a reasonable guess and either 0666 or 0644 the files (add the execute bit for folders). Unless you have a problem with the default or understand the consequences of changing this, I probably wouldn't override the default behavior.