If it were me (and had the time), I'd be running a query to see who's voting all of these 1's. I've seen way too many 1's myself and think there are a few people just being jackasses. Hopefully there's enough legit votes to help balance things out. RARELY would I rate something a 1 myself, it'd have to be the worst pile of dung ever really. The problem is as already stated, ratings in general. People have NO CLUE how to rate something.
Read some Amazon reviews on any product. They'll rate it a 1 because it arrived late, which has nothing to do with the product. Or they had a problem with the seller, so they rate the product (not the seller) poorly, etc. Not much you can really do about idiocy, if we could, there'd be a lot fewer people on the planet.
Ohh and... I like Shelly's idea, it's what was basically done with a vB mod (that really rocks BTW), example: http://www.dragonbyte-tech.com/f4/thread-ratings-v1-0-2-released-56
Look at the upper right corner of the thread. Anyway, it's really nice and for a ratings system is perfect IMO. Maybe more than xF needs overall, but still pretty nice. =)