That's one part of it.. the main is WHAT kind of file(s) are you expecting to be uploaded. If images, then I would typically NOT allow any any larger than 10-12MB uploads (of course, it then depends on what your XF max size is set at).
The other aspect is if you are allowing uploads of actual files (think ISO's, .ZIP for programs, etc) then you may need larger. As I said, you can't just use a blanket number as some would have you believe. You HAVE to know what type of files you are allowing to attach/upload and what the maximum potential sizes may be.