I take no pleasure in writing this. David Braben developed one of the essential games from my childhood. Braben was the man behind the classic Elite, the way ahead of its time space trading sim which was released in 1984. I was 12. All of these open world games where you have freedom to do what you want, when you want, go back to Elite. I spent more time playing that game than I care to admit.
But Braben said something really silly earlier in the year and tried to “clarify” his thoughts at the BAFTAs in London.
Yes, this is another Metacritic story. I don’t post this to give David a virtual noogie, although he clearly deserves it, but I really want the readers of our little ol’ blog to understand where I’m coming from when it comes to Metacritic and how people look at it and some of the insane ideas that people inside this industry come up with when it comes to the aggregate site and how it relates to people like us who are trying to do a job.
So off we go…
Before Braben backtracked, his original statement was this:
“Most reviewers are excellent at what they do, and it is a very hard job with, frankly, little glory. As an industry, there is something we could do to recognize this – effectively a Metacritic for reviewers.”
So far, nothing too crazy. In truth, I have always found it unnerving when reviewers don’t take their job seriously, and I know plenty who don’t, or when a game site assigns a review to someone who isn’t qualified to do it. When you read a review that starts, “I normally don’t play/like this genre, but here’s my 800 words on the subject anyway. You can trust me. Really. (Hey how many points for a made basket again?)
When those reviews get lumped in on Metacritic with those from qualified writers, it’s rather unfair to the reader and to the game. So go on David, you have my attention.
“The best reviewers give spot-on reviews pretty soon after a game is released. They do not wait to see what others say, but nevertheless consistently come very close to the final average score. There could be a prize for the best each year. Don’t forget – this is not intended to influence reviews – just to encourage and reward consistency – as it is not a high reviewer that gets the reward, it is the one that gets the best result.”
Whew, a lot to digest there.
First off, let me explain how this works.
The quality of the writer has nothing, at all, to do with whether a review is posted first. We have all read reviews from major websites that are published on day one that make us turn our heads like a dog that hears a funny noise. The size of the website has more to do with the speed of the review than the awesomeness of the writer or freelancer. So that’s just wrong.
In addition, there is never, ever, a “right” score for a game. Ever. We’re talking about a lot of variables here: the scale of the website, the influence of the writer, so on and so forth.
Example.
I gave NCAA Football 10 a “B-” grade, which in truth was being generous. This was turned into a 67 on Metacritic. That review was posted soon after release and I stand by that article 100%. I feel I was fair, honest, and wrote my ass off, thank you very much.
The overall score for NCAA 10 was an 83.
See here: http://www.metacritic.com/game/xbox-360/ncaa-football-10/critic-reviews
See that 67 at the bottom?
Hi mom!
If you want to use the Braben System of Reviewer Judgment I’m just some kook who should get a real job. I’m also apparently an idiot because I missed the average score AND an opportunity to earn a prize!
Braben basically wants reviewers to conform to one big, easy to digest opinion and if you review a game a week or two late or you don’t agree with the masses you’re not good at this whole “being a critic” thing. Bad idea, David. On a whole lotta levels.
Apparently, what sparked this was, according to Eurogamer:
Braben’s suggestion followed musings over the problems that face developers of children’s or casual games. He argued it was often difficult for such a title to get a fair, balanced hearing from ‘core’ orientated reviewers.
I think that’s fair — to worry about that sort of thing I mean. I know I wouldn’t want Barnes reviewing my brand new fluffy animal Candyland game either. I shudder at the thought. But this isn’t the answer, either.
You want to get your kid’s game out to the right reviewers? I have a suggestion:
Ask the editors.
We listen. Honest.
If you have a niche game, and developers of wargames have dealt with this for *years*, you ASK if they have anyone on staff who has familiarity or a real interest in the product. If you just blind send a game to a website there’s a good chance, if it’s a kid’s game or a movie tie in game it will get assigned to the “B Team” — that’s for the new guys, or the guys editors rarely give AAA reviews. (Yes, pecking orders exist; it’s how Brandon gets all the bad DS games.)
Do you see reviews of niche games like text based sports games, or hardcore wargames on every site? Of course not because you have to have faith that the website will review it with some level of competency. If you don’t feel that way — don’t send them a copy. Done and done.
But “grading” critics based on how close they follow the average score on Metacritic and on how quickly they post reviews is a terrible, terrible idea.
Now onto the backtrack.
Braben tried to qualify his statement by saying:
“I think a lot of people assumed I was criticising journalists, which wasn’t the intention.”
No, that’s not what I thought.
“What I was really saying was that actually, it’s a really hard job. Because if you want to try and make your reviews stand out, you end up producing something that, with time, will make you look silly. So, in actuality, hitting Metacritic early on is a challenge, and I am impressed by the writers who manage to do it. Essentially, yes it’s a very hard job and I apologise to those who either misunderstood what I said or disagreed. I’m happy to debate with them if they want at some time!”
Anytime, Mr. Braben.
(Elite was awesome btw! xoxoxo)
Lastly, Marc Doyle, co-creator of Metacritic and a man with whom I have exchanged several emails over the past year, and a guy I happen to like quite a bit, said this which made me smile:
“Penalizing a brilliant critic who happens to utilize the lower end of its publication’s scale more often than a middling critic who never gives lower than a 6/10 doesn’t make sense to me.”
Good on you, Marc.
Something I have noticed through time, is that some critics I agree with, and others, I do not. I will ALWAYS put more weight in the reviews by guys I see eye to eye with, over those I do not. The problem, it seems, though, is the author’s name is always in very little text. Let them stand out. EGN (I think) used to show pictures of the guy reviewing, so you knew at a glace who it was. Understand the reviewers, and people will get better experiences. Though, sometimes, you still get stinkers like Call of Duty Black Ops that is hopless on my machine… and no way to know that ahead of time, since I am WELL over minimum requirements.
I can’t believe anyone honestly puts stock into Metacritic. This is a site that simply doesn’t exist in my world, it’s a site that I do not care about, along with the hundreds of other reviewers who’s voices it represents. I care about a handful of reviewer’s opinions, and I read them. I have never, nor will ever, respect “the General Consensus.” The General Consensus brought Germany the Third Reich. The General Consensus brought the understanding that the world is flat – before a small, intelligent cadre changed that thought. The General Consensus said that Knights of the Old Republic II was a mediocre game with a weak plot. I’ve learned that people, as a general body, are usually wrong about a given subject. Think about how easy it is to agree with everyone else. Think about how easy it is to be just like all the others. Any dead fish can swim downstream.
Mr. Braben’s suggestion, and for that matter his worry, seems strangely resistant to the definition of ‘meta’ yet oddly promotional of Metacritic as a benchmark for rating games. The idea seems to be that a niche game won’t do well if thrown to the sharks who swim in the public pool, which is probably true – but Metacritic seems to be designed to cover the public, or certainly the mass consumer, opinion. He appears to be essentially alleging that Metacritic does its job too well for niche games?
I have to wonder, however, if the effect of Mr. Abner’s suggestion isn’t already promoted by the basic self-interest of reviewers and reviewing companies. It doesn’t make very much sense, for example, to review a game that you or your primary readership are not interested in. If a company does indeed have an editor with an interest in strategy games, then, it makes some sense to presume that said company would have a review on Metacritic if indeed the game was reviewed by that company at all.
A full test of this hypothesis would require a comparison of the companies who reviewed a popular, triple-A title and those who reviewed a more niche title, searching for similarities and then further comparing via company size. A basic run of this test, limiting itself to the companies listed upon the first pages for the reviews for Dragon Age II (the triple-A title) and Victoria II (the niche title), reveals at least four companies who reviewed both: PC Gamer, Gamer.no, Cheat Code Central, and GamingXP.
Metacritic ultimately acts as an amalgamation for all the reviews of a game that it can find, and I submit that the result of the admittedly basic test performed suggest the existence of a natural self-selection mechanism for niche games – simply, if a review source isn’t interested in it, it won’t review it. This seems borne out by the percentages – Victoria II got 65% of the total reviews that Dragon Age II did, but only four companies’ reviews made it to Metacritic for both games.
~ Matt
I expect I’m in the minority, but I almost never use reviews written the week a game comes out to make purchasing decisions. Too often, they’re reactions to the game’s publicity campaign rather than the game itself. For games with a strong community component–MMOs especially, but also anything with a lot of user-generated content–a massive part of the game experience just isn’t there on release to be reviewed. And a bunch of games coming out now have significantly different experiences on replay.
What I want is substantive, thoughtful writing about a game you’ve spent some time with, not a quick yea-or-nay based on a couple hours’ play. I want to know that there’s still going to be a worthwhile game there in a year, and I’m usually perfectly willing to wait that year to find out.
To me, the score is the least important part of the review. If I use it at all, it’s because I’ve noticed a game has had overwhelmingly bad one. (I’m talking about Fs, 30s, or One….
… Out of Five.) It serves more as a flag of public perception for a game. If it’s getting all 10′s or 100′s or 40/40 scores, it’s probably going to be hailed as some kind of marvelous achievement. This is fine and all, but even a game that gets great scores doesn’t mean I’m going to want to play it. See: God of War. I actually incredibly dislike the God of War games due to the simple fact I just don’t find them fun. I do realize that God of War is indeed a good game, but that doesn’t automatically make it fun for me.
One thing I liked in some of the magazines of yore was where in addition to the main review, some of the other writers would give their impressions on the game. Usually you’d know what types of games those reviews liked, so you had an idea of how people other than that writer felt. Metacritic sort of lets you do the same thing, but its scoring system is all sorts of crazy, especially the way it decides scores if you do not give it a x/100 or x/10 type of score.
So I avoid the score. Encouraging use of the score is the last thing that needs to be done. I feel like the more important a score becomes, the less reviews will be read. People will just check the score, not read the reviews, and make their decisions based upon the Metacritic average. The types of titles I like are super niche (Handheld JRPGs/Tactical RPGs/Farming simulators that also have dating simulators and possibly also dungeon crawling in the spinoff), and exist in a sea of mediocrity. It’s hard to make decisions on which game to buy based on the synopsis. (well, not on the dungeon-crawling-farming-dating game. That’s a day 1 purchase.) I need reviews to help me in these situations, and I need to actually read them.
I cried at the end of Speed Racer, and when Rainbow Dash performed the Sonic Rainboom. I fell asleep during Schindler’s List because it was so (description here). People should never agree on what is good, because then the world would be boring.
The review process always strikes me as just a way to squeeze a few more dollars out of “B” titles, or ones that can’t succeed without word of mouth. Games like League of Legends, WoW, Fallout 3, will all sell themselves without a review score, because they give people what they want. Games that fail to do this, all scrabble for a 80+ score that they never deserved.
:p
Wouldn’t giving all strategy games to ‘Strategy Steve’ and FPS ones to ‘Shooter Seamus’ be creating a bit of a walled garden over time? I suppose you don’t want to give ‘Blood & Guts 3: The Evisceration of the Elves’ to the person that got into the business reviewing facebook games, but the ideal of Metacritic (for me at least) is more to find that reviewer you agree with or can appreciate and see all the ideas on the game from good to bad from a wide variety of people all presented in easy reach.If it’s truly any good it should generally shine through in some fashion regardless of who is behind the screen and that should be present in the text of any half-decent reviewer. Even when I disagree with a reviewer after all, I can usually at least appreciate their perspective on the game and can potentially see the games in a different light myself as a result.
Unfortunately, Metacritic has just become the gold standard ‘review’ in and of itself due to a number it reaches via relatively arbitrary means and the people that just use that number to the exclusion of all else. Until you change the mindset of people using it or make a better way of getting out that info I’m not sure there’s any solution to this. Sadly, the end result I can see is that all reviews start using a 100pt scale and then playing it safe by weighing things in the upper range as I’m sure they don’t already! …but more widespread even…which is depressing
It depends on the reviewer and the game.
Ideally you want to have more than 1 go to guy/gal for a genre, even specified ones.
That said you do not want me reviewing a flight sim. Or NASCAR game. My opinion on those games would truly be useless. And there *are* writers out there that will review anything under the sun if you let them in order to collect a check/free game. A good editor won’t allow that to happen, though, but MC isn’t the wiser when it does.
In a [perfect world there would be no scores at all or we’d all adopt the thumbs up/down system. (No waffling!)But it is what it is.
There are games for all of us that we just don’t ‘get’ and if you’re that far off the mark for the target audience you might not be the guy to get an opinion from.
On that note, I’ve always wondered why we don’t get children to do a co-review for children’s movies/books/games assuming you can find one that can form a cogent opinion and all. Or perhaps having everyone in the office play the same game with wildly different types of people to each see what they each get out of the same ‘base experience’. I think reviews themselves should be a bit more flexible on those fronts at times – i.e. the wonderful Shogun write-ups you’re currently going through. Works of art are generally much more than just the sum of their parts or the average of their score after all. I just wish the industry believed in that concept.