Let us explain.
Every December, the editors of Motor1.com take stock of that year's new car reviews. You get to experience this via our big roundup of the brightest and best vehicles we drove, but internally, we spend time looking for cracks in the system – not only do we analyze outliers in the actual review scoring, we also look at your concerns and suggestions, sent via email or shared as a comment on the site.
The result of this analysis is our annual rating system update. But our changes for 2021 are bigger than anything we've done before. Here are the changes we've made to our new car vehicle rating system and how they'll affect this year’s ratings.
For 2021, we're keeping all seven ratings categories (Design, Comfort, Technology and Connectivity, Performance and Handling, Safety, Fuel Economy, and Pricing). But instead of measuring every vehicle on the same scale, we've assigned a class and a set of competitors (which are always listed at the bottom of a review/first drive) based on size, function, price, and mission to every product on the market – there are 138 unique classes in total. We then made sure the scoring in all 138 classes had their own unique weighting for the seven categories.
The CUV/SUV segment is our largest, but we’ve managed to divide it up to account for the differences between similar vehicles, like the BMW X3 (Compact Premium CUV) and the X3 M Competition (High-Performance CUV). There’s a separate category for Chevrolet Suburbans (Mainstream Truck-Based SUVs) and Cadillac Escalades (Premium Truck-Based SUVs), while we’ve even broken enthusiast-focused vehicles like the Jeep Compass Trailhawk, Toyota RAV4 Off-Road, and Ford Bronco Sport Badlands into their own dedicated category, rather than lumping them in with other compact crossovers.
Some classes put more weight on Performance while others emphasize Safety, Price, or Fuel Economy – more often than not, classes weigh multiple categories differently. A mid-size mainstream sedan may earn the same seven scores as a mid-size luxury sedan, but the overall Verdict for the two will be quite different. And no, a higher score for one does not mean it's inherently better than the other.
Let’s revisit the X3/X3 M Competition as an example. If both received a seven out of ten in all categories, the standard X3, which is a very balanced package, would earn an 8.9 and the performance-focused M Competition model would return a 8.3. But if we increased the Performance score of each car to nine, the verdict changes to 9.2 for the X3 and 8.9 in the X3 M.
If we go a step further and remove one point from the X3’s Comfort and Pricing scores (the two categories Compact Premium CUVs put the most emphasis on), the overall score falls to 8.8. Those are some significant swings considering the similar category scores – just imagine what could happen with ratings scattered all over the place.
More Accurate, Understandable Scores
The result of this seismic shift in how we rate vehicles means we'll see higher scores across the board – the average SUV/CUV verdict on our old system sat at 6.6, whereas the new average so far is 8.3. But this isn't simply a case of a high tide raising all ships. We've been testing this new rating system since September and comparing it with scores on our old scale to ensure that the new verdicts are both more accurate because of this fine tuning and easier to understand, too.
One of the strongest criticisms of our old system is that vehicles we wrote positively about earned scores that seemed mediocre. A 6.2 out of 10, after all, is just 62 percent, or a “D-“ on the grading scale Americans are familiar with. With that in mind, we built this new system to more closely align with the letter grade readers recognize from their youth – exceptional vehicles will earn 9s (or an “A”), while average cars will get the Cs (or 7s) they deserve. Failing grades will happen, too, but the important thing is that our scale is shifting from the 4.0 to 8.0 range to a more understandable 5.0 to 10.0.
We're Ranking Everything
Another critical weakness in our old ratings system is that we didn't score vehicles until we could test them on our home turf via a week-long media vehicle loan for a proper review. That's changing in 2021. In addition to every Review, every First Drive we publish in 2021 will have a verdict attached. We'll reserve the full run-down of scores for our normal Reviews simply because our method of displaying those on the site is software limited (we're looking to change that), but you'll see ratings now at the top of every single First Drive going forward, and each of product will be eligible for our end-of-the-year roundup.
The biggest point of contention for you, the reader, has been fuel economy scores. Going along with our new class-specific score weighting, we've reworked the Fuel Economy portion of our system. Every vehicle class now has a target efficiency number above the current class average (for future-proofing purposes). We'll continue to judge gas-, hybrid-, and diesel-powered vehicles based on their EPA-estimated combined fuel economy. For plug-in hybrids and EVs, the target is based on range (until the EPA or some other entity starts sharing ratings of miles per kilowatt-hour).
This means that V8-powered muscle cars, truck-based SUVs, and all-electric CUVs have a different fuel economy target and that any vehicle could earn a 10 for fuel economy – but, that 10 may carry less weight, depending on the vehicle's mission. We're also rating vehicles based on best/worst-in-class fuel economy, their need for Premium fuel, and in the case of EVs and hybrids, for best/worst-in-class charge speeds.
Changes are coming to our Pricing and Safety ratings, too. In the past, the core of both areas was on the overall model range. We'd look at the real base price and judge it alongside the as-tested figure to determine the Pricing score.
For 2021, we'll be looking at the base price for the specific trim we tested, along with the as-tested figure – this will punish vehicles that require thousands of dollars of optional extras and benefit those that don't. It's a similar story on the safety front, where we'll dole out extra punishment for trims that don't offer active safety gear, rather than looking at what’s available on the entire range (although we’ll still note that equipment in the text).
In addition to these major changes, we've made a number of smaller updates in individual categories. For instance, ride quality is now a specific measure on Comfort scores, and we'll be awarding and subtracting points for convertibles with especially quick or slow roof mechanisms. We'll also consider the implementation of active safety equipment in Safety scores, so vehicles that ping-pong the car around a lane will lose points while especially well-integrated systems will earn them.
Our new rating system will take effect with this week’s reviews. As always, we encourage you to offer your views in the comments of this post or at firstname.lastname@example.org.