In this context, 0 means inherit so that's not entirely unexpected.
Even so, you'd be unable to override a more permissive integer value (e.g. a higher number or unlimited) with a less permissive integer value. So if it's set to 5 on the user group, and you try to set it to 4, 3, 2 or 1 on the node for that user group, then the most permissive value (5) will be the one that takes effect. Similarly, 0 wouldn't actually mean 0 in this context, but it would just mean "Inherit" so that would also resolve to "5".