Sure, if you trust them. And if you do trust them but don't want to run that risk? I want them to be able to ban users, not the access the admincp. Granted, there are some of them I would not care giving access, this is not extensible to all my moderators or the new ones. Also, some of us are more careful than the rest and have their admincp htpasswd protected and nobody should go near that, ever.
What is all this talk about they "only" have access to the Rebuild Cache option? That is not an only, that is a "huge" only. Rebuild search index is there. That is so server intensive that someone can reduce my site to a crawl just by playing with it. Might as well be "only" the option to turn the forum off. Has this even been reported as a bug?
So .. yeah, the current solution from XenForo is not enough not appropriate for my use case.