Yes, and no. I think this might work with .. some modifications.
If 5 random users add you to your ignore list / report you, then yes, it is a terrible terrible idea to ban the user.
But if 5 users, from a list hand-picked by the administrator, add people to their ignore list / report the content, then the user might be a good candidate for banning and I would say ban him first, ask later. If the administrator feels that those people are abusing their power they can just go and remove them from the elevated privileges list. That way, people would be careful of not involving themselves in some sort of pity war, short of loosing their privileges.
This is a soft approach to moderators. These members would not have the same responsibility / power as a moderator (closing threads, editing posts, moving content, splitting it, giving advice, having an official title in the board) ... these are more like a group of senior members, that are trusted by the community with special privileges in the forum. They might or might not be named (the administrator can choose to hide this behaviour, or they can advertise that John, Luke and Paul can get you banned.) For all I know they might never know they have the power to ban people. They are just ignoring and reporting.
I have been giving these ideas some thought lately, some kind of peer moderation. Especially when content jumps to thousands of threads a day, the moderators system does not scale. Sure, I can always add more moderators, but who is going to watch their actions as I do? Having some sort of validation done by the community could be certainly helpful. Maybe I don't trust a single person the power to ban people from the forum, but I might trust that 5 votes from a specific group would do.