You can allow different user groups to get a "bigger say" in the moderation. In group permissions under "can report content" you'll see a new option where you can assign a number for them. If you assign 0, the votes won't count at all towards moderation.