Sunday, February 26, 2012

The unadjusted rating scale and low ratings [Rating Editorial Part 2]

[NOTE: As of late 2013, I've removed numeric ratings from all my posts, so there will be some references in this post to a rating system that no longer exists. My apologies for the confusion.

This is the second part of my editorial about review score ratings. While part one looked at the "cursed" 7/10 and how to avoid it, this article will discuss the difficulties of using an unadjusted (i.e. non-inflated) rating scale, focusing on the lower side of the scale.

My face when I'm deciding what ratings to give.

[Note: for the purposes of discussion, I will restrict ratings to "shows," so I don't have to say "show/game" and "watch/play" all the time.]
     So, let's just jump right in, shall we? First I guess I'll talk about the difficulties with above average ratings, the 6-10 range. I may as well start with 10. This is perhaps the most misunderstood rating of all of them. This is because everybody has different opinions on what makes a 10/10. Some people feel that a 10/10 should be reserved only for those shows that are perfect and flawless in every way. These people generally never give out any 10/10 scores, because nothing will ever be totally perfect. Other people believe that a 10/10 is a "masterpiece," and that only those exceptional (but not necessarily perfect) shows should get it. They are much like the group above, only a little more lenient in giving them out. Then, there are the people who give 10/10's based on how much they enjoyed the show. If they absolutely loved watching it, then it gets a 10/10. Then there are the people who give 10/10's based whether or not the show was well-made, disregarding the overall quality and focusing on production values, then there are the people who rate based on personal feelings, et cetera, et cetera. All this to say, there are a lot of standards for what makes a 10/10. This can get very confusing for a readers, because there no common standard. They have to jump from one review standard to the next, and the jump is not always successful. The result of all this is that 10/10 are extremely hard to use effectively on an unadjusted scale. Heck, on any scale. One has to very clearly state what their criteria for a 10/10 are when they give it, otherwise people will assume they're using the standards of those other reviewers they've seen. Often, this leads to a misunderstanding between ratings, and subsequently leads to an unsatisfied reader.
     Next is 8/10 and 9/10. These two are actually the easiest of all the ratings to use. The reason is that they are pretty much the same no matter where they are used. An 8 is almost always "great," and a 9 is almost always "excellent." There are very few varying definitions on what an 8 or a 9 out of 10 mean.
     Then, the cursed one, the 7/10. This one is actually fairly straightforward to use as well. One simply has to explain that a 7/10 implies a higher quality than a non-inflated 7/10 does. There's not much more to it than that.
     Now we're getting into more murky waters. A 6/10 is difficult to use, because most everyone thinks that 7/10 is "average." Thus, they immediately think of a 6/10 as "below average." Since this is the opposite on a non-inflated scale, and a 6/10 is actually above average, it can be tricky to use. You have to strongly defend the reasons why it's good, but you can't overplay them, otherwise it will sound like it's a 7/10. As I discussed in part one, many people tend to apply school grades to media ratings. This means that a 6/10, or 60%, is a "D." D's are kind of the cursed letter grade. They're liked a mark of shame, even more so than an F. A "D" means that the person almost had it, but not quite. It's like telling them that they sort of understood, but didn't try hard enough and that's why they didn't get the passing C grade. Apply this to media, and people start thinking "Oh, this show is just 6/10. That means it missed the mark, it has wasted potential, it's not worth it." Trying to communicate that this mindset should be abandoned is difficult, to say the least.

"6/10, huh?"
     Now we're moving onto the 5/10, neither positive nor negative. This rating is not particularly hard to use because of the score. It's fairly easy to explain that something is "totally average." In fact, a 5/10 can actually be helpful to the rest of the scale, because calling it average clearly shows that your rating system uses an unadjusted scale. Instead, 5/10's are hard to rate because of the material. When you rate something 5/10, you're calling it a perfectly average show. In other words, you're establishing the standard that any "good" show has to pass. This is difficult, because it can trip you up later on down the road. If something doesn't surpass the standards you've established for yourself, but you say it does, then it's like sticking your foot in your mouth. 5/10's establish standards for everything else. This is why you don't see a lot of these. No one want to be tied down by the rope of "I have declared this to be the perfect average." They prefer to leave what exactly "average" is as an ambiguous identity, and not without reason.

You looking like this yet?
     And finally, we can get to the main focus of this editorial, the lower side of the scale. These are the most difficult ratings to use, and I'm afraid I don't have any wisdom or recommendations to share. So what is it that makes 2-4/10 so dang hard to use? Well, in the eyes of the consumer, they all mean the same thing. It doesn't matter how bad something is, it's still bad. People are just as unlikely to watch a 3/10 show as they are a 2/10 one. And that's the problem; there is zero reward for not using these interchangeably. What, do people care if something was a car crash instead of a trainwreck? Probably not. Both are things they want to avoid, so they will. This is where our ethics and principles as reviewers come in. Despite the fact that no one cares, despite the fact that it's all the same in the readers' (and maybe even our own) eyes, we still have to try to use these intelligently and fairly. Here we are getting into the dark gray area that is varying levels of bad. It simply wouldn't be fair to say a show that deserves a 4/10 is actually a 2/10. No one really wants to see either, but it simply wouldn't be fair to the show or yourself to incorrectly rate it. If you went through the trouble of reviewing the show, you shouldn't destroy the meaning of that review with an incorrect rating.
     Last we have the bottom of the scale, 1/10 (or 0/10, if you do that). This one is a much like the 5/10, in that when you give this rating out, you're establishing that show as the standard of terrible. If something worse gets a 1/10, then people will ask why the first show wasn't higher, and if something better gets a 1/10, people will ask why it isn't higher. It's not quite as difficult to use as a 5/10, but it's close.

     There is another reason why non-inflated rating is difficult. This reason is that, like with school grades, we have a tendency to rate down from 10. This creates the problem that we have now. Around 6/10, we feel that things are far enough away from 10 that they're no longer worth our attention. Rating on an unadjusted scale means that you to rate both up and down from 5. This is hard, both to communicate and to practice.

Trying to rate intelligently is like trying to find Russians.
     Rating on the unadjusted scale is a long and sometimes unrewarding process. You have to constantly quantify your ratings, and you have to spend a lot of time evaluating the material and determining whether it's good or just "okay." Perhaps the hardest thing, however, is avoiding the urge to rate unfairly. Spending lot's of time determining just how low a rating should be is rarely rewarding work, but it's work we have to do nonetheless. In all honesty, rating something low is more a matter of self-appreciation then reader-appreciation. When I give a low rating that I thought about and evaluated for a long time, I can pride in the fact that I didn't give an unfair rating. Still, using low ratings is an art that no one, including me, has mastered. Perhaps no one will. But we can try.

What are your opinions on the unadjusted scale and how to use low ratings? Please, share your thoughts!


  1. It's interesting how you covered the biasses of our perception when it comes to looking at the numbers. Some viewers who are too lazy to read the full review, and only care about the score. Whether the ratings are adjusted or not, the best way to communicate ones scale is to explain this in a simple yet striking word(s).

    So do we really need a standardized rating system? Individually, I believed yes in order to preserve the coherence of our reviews. As a whole, I don't think so. I think it's always up to the writers how they're going to make their reviews interesting and believable.

    I'm not really fond of reviewing and rating anime, that's why I seldom write one. But I believe that numbers can easily communicate our enjoyment to our readers as long as it's well defined.

    1. Thanks. If possible, I like to explore as many sides of a situation in as neutral a light as possible. My success varies, of course, but I figure it's worth trying.

      I agree. If everybody had the same rating standard, the blogosphere would actually be kind of...boring. I mean, it can certainly be a little difficult for readers to transition from one style to the next, but as long as there is consistency within each site, then it's okay.

      Same here, actually (just look at the first image). Originally, back before the blog even existed and I wasn't really sure what I wanted it to be like, it was going to be a strict game review site. Since game reviews sort of include number ratings by default, I never really thought there could be anything else. By the time I started reading other people's blogs (like yours) and found out that very intelligent and thoughtful yet opinion-conveying reviews could be written without the use of numbers, I had already started reviewing anime with number ratings, and I figured it would be a bad idea to abandon them at that point. So far, they've actually worked pretty well for me, so I'm hoping it continues...

      Thanks for commenting and checking out the blog! I appreciate all the feedback I can get!

  2. I was reading your back-and-forths with that commenter on your last article, and really wondered: what's the point of 'objective' rating? How can we be 'fair' to anime? Does an anime that you personally gained no entertainment out of (even though it had great plot, moving characters, stellar music, and wonderful execution) really deserve a 90/100? Actually, does it deserve any more than a 0?

    Just some food for thought.

    P.S. Yes, I am stalking your blog because :bored:

    1. Well, we're getting into the idea of why I make reviews in the first place. See, this is gonna sound a little odd considering what I read and what I enjoy, but I believe a review that is all personal, subjective opinions is. . .not worthless, but also sort of lacks a point. I mean, yes, I want to share my thoughts with the others, but the point of my blog isn't to make others listen to me. Now, a highly opinionated review is okay, I feel, as long as you explain *why* you hold those opinions. Now, I like the critiquing process personally, but the main point in my opinion is that it's there to help other people determine if something interests them.

      See, if you get down to the point Raggers was going for, then I feel objectivity is a little pointless. No matter how far you break it down, objectivity *will* be lost sooner or later. The rule of thumb I've found with reviews is to give it a range of about 1 point less than or greater than the actual score. . .but I digress. See, I want my reviews to be *helpful*. I also want to talk about why something is good or bad, of course, but I want my readers to walk away (figuratively) with *something*, preferably an idea of whether or not they should watch a show. If it's something good, I want to share it with them. If it's something I couldn't stand but I feel is very well made (like the PC game Blood, for instance), then by all means, I want them to go out and give it a try. So yes, some great anime that didn't do anything for me at all but is really, really good *does* deserve a 90/100, as far as everybody else is concerned.

      At this point I feel like I'm just spouting words, but I feel there's something I should close with, to add some clarity to that last statement. The ratings on my reviews, and the ratings I personally assign things, are not necessarily the same. I feel I wasn't so good at it early on, but I can enjoy something, realize it's a critical (as in, once critiqued) mess, and tell people why it's a mess while still enjoying it personally. As far as *I'm* concerned, it may very well be a 0/100, but as far as my *readers* may be concerned it's a 90/100, and I like to think I can make that distinction to a certain extent. Though usually in cases where the two scores are so far apart I just don't review it. *shame*

      P.S. I'm honored, Mushy. Really, I am.

      P.P.S. I forgot to say this last time, but thanks for dropping by and commenting!