Scoring systems for rating cigars, beer and other items vary greatly. There are 5 points scales, 10 point scales, 100 point scales and then a combination of other rating type systems. Each of them have their pluses and minuses. Primarily focusing on the point rating systems, here are some thoughts on the three most popular systems.
5 Point System: This system is a bit limiting with only a 5 point range to convey great to bad. Some 5 point systems then utilize half points to increase the range of scoring, but in that regard, why not just use a 10 point system? For most of the people that use this scale, you’re going to find the majority of things scoring in the 3 to 5 point range.
10 Point System: We tend to feel that the 10 point rating system works the best. A 10 point scale is easily understood and gives a good range between the highest and lowest scores. For most of the people that use this scale, you’re going to find the majority of things scoring in the 7 to 10 point range.
100 Point System: This is one of the most popular scoring systems as it’s used by some of the high profile review organizations/publications around cigars, wine, beer and spirits. Most people have seen it and understand the concept behind it. This is the scoring system that we dislike the most. The reason for our dislike is that on a scale that has the greatest range of options of scoring, most entities that use it only utilize 10% to 25% of the scale. If most everything that gets rated ends up between a 72 and a 97, why have all of the other numbers? Why don’t you see anything scoring a 35? Is there nothing that deserves that score, or if there is, is it just not being published? All questions we have that we’ve yet to find the answers to. If you know or have theories, we’d love to hear.
Along the lines of the usable scoring range mentioned for the 100 point scoring system, another issue with scoring systems is how the scores are generated. If scores are calculated based on points assigned to each category, then if you add up all of the lowest scores for each category, that is the lowest score possible. If that score is not the lowest number in the scoring scale you are using, then the range is skewed. Why have numbers on the scale if there is no possible way to achieve them?
We wanted to use a scoring system that utilized an entire range and not just a subset. We did some basic research to see if we could find an easily detailed system that provided what we were looking for, but we weren’t able to find anything that matched. This isn’t to say that it doesn’t exist and that we haven’t created a replica of a system that already existed and we just weren’t aware of it. Here is an explanation of the system in its basic form:
A 10 point scale will be used for the total score.*
There are seven categories of which only six contribute to the scores; Pre-Light Experience, First Third Flavor, Second Third Flavor, Final Third Flavor, Burn, Draw and Overall Experience. Each category is weighted into how much it contributes to the final score as we don’t feel that all categories should count evenly. A breakdown of the weighting is as follows:
Pre-Light Experience – 0%: Even though we describe our experience in this category and give it a ranking, it does not factor in to the score. In our opinion, the look and smell of a cigar before being lit has little effect over whether or not we’ll enjoy the cigar. This is similar to the “Go, over show” concept that people into cars will understand.
Each Thirds Flavor – 21%: We feel flavor should be the biggest factor in the rating of a cigar, and that is why the weighting is so high. In adding the thirds, they contribute 63% of the total score.
Burn – 6%: Although burn can have a major effect on the experience, especially if it just won’t burn, we think too many people get caught up on whether the burn line is razor sharp. To us, the burn can be wavy. We don’t usually go back to the lighter unless there is a drastic issue.
Draw – 9%: Draw is fairly important to the scoring as it can have a huge impact on the experience. If you have a draw issue too far in regards to being loose or tight it will have an impact on flavor and may overheat the cigar based on how you try to deal with it.
Overall – 22%: This category allows for taking the entire experience into account. Specific categories individually may have done really well or really poorly, but when everything comes together, how was the experience?
* For reviews of cigars that were released in the calendar year of 2016 and earlier, we used a weighting breakdown of 20% for each third’s flavor, 9% for burn, 12% for draw and 20% for overall. The reason for the change was that we found that the construction scoring artificially inflated the overall score.
There are six categories that contribute to the scores; Appearance, Aroma, Flavor, Balance, Finish and Overall Experience. Each category is weighted into how much it contributes to the final score as we don’t feel that all categories should count evenly. A breakdown of the weighting is as follows:
Appearance – 5%: Is the beer visually appealing? Does it make you want to immediately start drinking? Does the color and viscosity match the beer style? How’s the carbonation? These are some of the criteria that factor into the appearance, albeit a small amount of the overall score.
Aroma – 10%: The aroma of the beer should also match the style. Are there any notes that seem off, or is the aroma something that enhances the overall beer experience?
Flavor – 45%: We believe flavor is the most important part of the beer experience. The aroma and appearance can be unappealing but ultimately the beer needs to have great flavors for you to enjoy it.
Balance – 10%: Balance is an often overlooked aspect of beer tasting. If you have a style that is very alcohol forward, it can completely run over the other flavors you’re supposed to be tasting. You also don’t necessarily want one flavor being the only flavor you can taste.
Finish – 10%: The finish of a beer completes the flavor experience. If the style carries nicely into the finish, it makes you want to take another sip. If the finish isn’t very good then you’ll find yourself waiting much longer between sips.
Overall – 20%: This category allows for taking the entire experience into account. Specific categories individually may have done really well or really poorly, but when everything comes together, how was the experience?
More Detail on Scoring of All Items
Each category has 7 rating options. This is where things can get a bit confusing. A 10 point overall scale is being used even though each category has 7 selectable ratings. Based on the category weighting, each of the 7 rating options carries a numeric value. Once all of the categories are scored, they are added up to get the final score. There are lots of numbers after the decimal point involved, but the overall score is a number and two numbers after the decimal point. If you wanted to really get down to it, rather than saying we are using a 10 point scale, you could say we are using a 1000 point scale if you dropped the decimal point. The reason for us using a couple of numbers after the decimal point is that with the 10 point scale, it won’t take long before you have two items that have the same number. Are they truly equal or are their slight differences that makes one a bit better than the other? We’re hoping the additional numbers help show those differences. You will see the score from each reviewer individually, but not an averaged score across all of them. The reason for this is that we feel the average score will be looked at as the score to go with rather than looking at and understanding each individual reviewers score. You should associate more with the score from the reviewer that matches closest to your palate, and an average score takes away from how likely you will reach that same conclusion.
Here is where we try to walk the walk. If a particular item scores the lowest ranking in each category, that score would equal out to a 0, which is the lowest number of the 10 point scale. Now, whether or not an item that actually earns that score is reviewed is to be determined, but in this system it actually is possible.
The guidelines for the scoring system are as follows:
10 Points: An amazing product, maybe the best we’ve ever had or at least a top 5 or 10. Probably not going to see many of these.
5 Points: An average product. There are lots of average products out there and this is where they should land on the scale, not as a typical 7 on a 10 point scale or an 85 on a 100 point scale.
0 Points: This is a very bad product. Similar to a 10, there probably won’t be any true 0’s, but a truly bad product would earn this score or close to it.
Being able to somehow correlate the cost of an item into the scoring is a tricky subject. There are some sites that include the cost of the product in their scoring system. We feel that adds another layer of subjectivity to the scoring in regards to how that person determines value, so we wanted to take a different approach.
Our scores are purely on the flavor and performance of the item and we want the score to reflect that, regardless of price. Below our scores, you will see a value which is simply the suggested retail price of the item divided by the score that each of us gave the item. This is what we call Cost/Point.
In its most basic form, typically an item with the lowest Cost/Point would be the best value, but that is not always the case. For example, a $3 cigar that scored 4 points (below average) would have a Cost/Point of $0.75. Conversely, a $10 cigar that scored a perfect 10 would have a Cost/Point of $1.00. In this case, the lower ratio doesn’t necessarily reflect the better value. It really depends on if cost is the primary factor over flavor and experience for you.
We suggest that you compare the item score and cost when looking at the Cost/Point to help you determine which products may show their value to you.
This is the system being used for our ratings. We are open to feedback and comments, so please, let us know your thoughts.