Even if the XFMG is not viewable to certain user groups per permission settings, the Statistics Widget Block in the Sidebar is still fully visible with all data. It should not be visible if the permission "View Media" is set to no.
Let me add that I realize that XenForo is made for public boards in focus – which makes it hard for complete private boards – as I do setup. I wished XF could be made completely private with one checkmark. Now I have to work with conditionals and hide individual template elements manually.