In October of 2017 we published an article called “When a 90 is not a 90” which explored how various review entities that utilize the 100 point rating system stacked up from 2015 through 2017 to see what each looked like for average scoring, most frequent scores, etc. to determine how much score inflation was occurring. This article just reported data collected.
A month later, we published a follow up article which included opinions about each of the sites and the results from the data collection.
In December of 2018, we posted an article revisiting the sites a year later along with this article in December of 2019 revisiting the sites two years later. These revisits tracked their scores over those years to see what changes, if any, had occurred. We also included some new sites that weren’t in the previous articles as well as removed some sites that stopped publishing or weren’t publishing frequently enough to have meaningful data. These articles were a mix of collected data and opinion.
This year, we are following up with data from 2020 (December 2019 through November 2020). This year, no new sites were added and two were dropped. This article will be a mix of collected data and opinion.
Revisiting
Cigar Aficionado
Data: The average score over the last 12 months has been 89.01 compared to 89.03 last year and 88.81 the previous year, so in between the two previous averages. In regards to the most frequent score given over the previous 12 months, the number dropped back down to 89 when it had been 90 the previous year. 90 was the next most frequent score with 88 being the third most frequent. Cigars still need to score a 91 to start differentiating themselves from the pack in their scoring range, but that gap is decreasing.
Opinion: Cigar Aficionado continues to maintain the consistency in their scoring over time and this last year has been no different. They are also one of the few review sources that uses a decently spread out scale. Regardless of what you think of their review process or what they review, you can at least take solace in the fact that they keep the scoring consistent. The bell curve showing their scoring over the year is about as clean as it gets.
Blind Man’s Puff
Data: The average score over the last 12 months has been 90.9 compared to 91.05 last year and 90.74 the year prior, so a slight decrease from last year and a slight increase from 2018. In regards to the most frequent score given over the previous 12 months, the number remains at 90 as it was last year while 2018 was 91. The score of 91 was the second most frequent given and 92 the third most frequent. Cigars still need to score a 93 to start differentiating themselves from the pack in their scoring range, but that is starting to get a bit blurred as the gap at that score to the rest of the frequent scores is decreasing.
Opinion: It’s nice to see a slight reduction from the increases the previous two years. They appear to still be cycling through new reviewers, and with this I think it is harder for them to maintain a structure of how scoring is actually performed. Overall scoring still appears to be inflated as the average and most frequent scores are 90+. The graph showing the scores for the year shows a bell that is skewed to the higher side of the average which supports this. As I continue to say, the site could use a re-centering of the scoring by modifying the scoring calculation so that non-flavor categories don’t have heavier weighting than flavor along with revisiting how reviewers submit their scores or re-educating the reviewers on the existing system.
Cigar Coop
Data: The average score over the last 12 months has been 89.35 compared to 89.96 last year and 90.78 the previous year, so slightly more than a half point decrease over last year. In regards to the most frequent score given over the previous 12 months, the number remains at 90, as it was last year, with the year prior being 91. The score of 89 is now the second highest with 91 being the third. Cigars continue to need to score a 93 to start differentiating themselves from the pack (more on this in the opinion section).
Opinion: The site continues to make modifications to its scoring system in order to try to drive a lowering of scores which the changes from these last two years have proven to work in that regard. The drawback is that the scoring window has become so narrow it’s very hard to see a score variance to really see the difference between cigars. You can see how narrow the bell is in the graph. What has occurred is that no cigars actually differentiate themselves in this scale, so it’s hard to convey which cigars are special. This definitely needs addressing and will be another tough undertaking for the site.
Cigar Dojo
Data: The average score over the last 12 months has been 88.92 compared to 88.56 last year and 88.95 the year prior, so over a quarter point increase from last year and right on par with the year prior. In regards to the most frequent score given over the previous 12 months, the number has jumped up to 91 when it was at 89 the two previous years. 89 and 88 are tied for the next most frequent score with 90 coming up next most frequent. The distribution of scores remains fairly odd as you can see by the graph, and it appears that a cigar needs to score a 93 to start differentiating itself from the pack.
Opinion: This was quite an odd year for scoring on the site. They have a large team now, but the total number of reviews continue to decrease (just over 1/week on average). The site had been decreasing its scoring the previous two years, but had an increase this year. With the low number of total reviews, the increase could just be due to the cigars that were selected for review, so it’s tough to pinpoint any solid reasons for it. It will be interesting to see how the next year goes to determine if the volume of reviews continue to decrease or if the volume picks back up.
halfwheel
Data: The average score over the last 12 months has been 87.32 compared to 87.23 last year and 86.62 the year prior, so just a slight increase over last year. In regards to the most frequent score given over the previous 12 months, the number 88 is the same as last year and the year prior. The second most frequent score is now 90, with the third most frequent being 87. Cigars continue to need to score a 92 to start differentiating themselves from the pack which is inline with how things were last year.
Opinion: In regards to the average score, the site has seen small to medium increases over the last three years, and this year is the highest average score of the six years that we have data for. With how wide the scoring range is (the widest of any on this list), as I said last year, it just takes one or two fewer cigars scoring very low to change a bit of the average. The scoring distribution is a bit odd and there’s not a fluid bell curve here and there is a dip around the top of the bell that you wouldn’t expect. It’s something to keep an eye on to see if the increase continues on in future years to determine if this is more of a trend than an anomaly.
The Cigar Authority
Data: The average score over the last 12 months has been 93.05 compared to 92.78 last year and 92.22 from the previous year, so an increase of slightly over a quarter of a point from last year and an increase of over three quarters over the previous year. In regards to the most frequent score given over the previous 12 months, the number dipped back down to 92 after it was 93 last year and 92 the year prior. The second most frequent score is now 93 with the third being 94. Cigars continue to need to score a 95 to start differentiating themselves from the pack as they did last year.
Opinion: The annual scoring average increase continues for the site as it has for all of the six years that we have data for. The graph shows how lopsided the bell is on the higher scoring side. With only three reviews out of 102 scoring less than 90 (all 89’s), it’s really hard to try to translate how these scores match up to many of the site on this list other than just subtracting a few points from the scores right off the bat to try to make sense of them. It’s probably just best to accept that the reviews are a method for the retail arm of the site to sell more cigars and leave it at that.
Tiny Tim’s Cigar World
Data: The average score over the last 12 months has been 92.51 compared to 92.03 last year and 90.28 the year prior, so an increase of nearly half a point over last year and almost two and a quarter points from the year prior. In regards to the most frequent score given over the last 12 months, the number is now 94 which is a point higher than the previous two years. The second and third most frequent scores are 92 and 93. It’s still really tough to determine where cigars begin to differentiate themselves as two highest scores are more frequent than the range of lower scores, but you might be able to say it’s 95 or 96.
Opinion: Shortly after our original scoring article was published, Tim chose to revamp his scoring system. The results were very interesting as the average score had dropped but the most frequent score given had increased. There was an increase last year and then another increase this year which is now the highest average we’ve seen since we’ve been tracking the site. I’m not really sure how to explain the increases the last two years unless Tim feels the cigars he’s been reviewing the last two years are just getting better. The graph shows a bell that is very interesting as it peaks so far to the right but is weighted so far to the left.
Stogie Press
Data: The average score over the last 12 months has been 91.66 compared to 92.40 last year and 92.48 the year prior, so nearly a three quarter point decrease over last year. In regards to the most frequent score given over the last 12 months, the number is 92 which is one less than the 93 it was the previous year. The second most frequent score is 90 and that was only one less review than the most frequent score had. The third most frequent score was 95 and this was only three fewer reviews than the most frequent score had. With the wide distribution of the most frequent scores, cigars now need to score a 96 or above to start differentiating themselves from the pack.
Opinion: This site is the youngest site in terms of how much scoring data we have and this additional year of data helps to bring a bit more insight. Even though there was a significant drop in the average score, the site still scores quite high. This year had some cigars score quite low which I believe is what caused the decrease in average. We’ll see if some of the lower scores continue, but in the end, as you can see by the graph, the scoring is quite skewed to the higher end. This site remains one of the most inflated scoring sites of the group.
Overall
Comparing average scoring of all of the sites between their most recent scoring and last year, four sites increased their average and four sites decreased their average. The increases for The Cigar Authority and Tiny Tim stand out the most as they are large increases and have a track record of them. I’m very interested to see if they continue to go up year after year. The increases for halfwheel and Cigar Dojo aren’t that big and are within range for some standard fluctuation. The decrease for Stogie Press seems more like the inclusion of some low score reviews more than an actual change the scoring itself, so it’s something to monitor. The decrease at Cigar Coop is significant but with the very narrow scoring range now, seems to have caused a negative byproduct that needs to be resolved. The decreases at Cigar Aficionado and Blind Man’s Puff are slight variances and are likely just standard fluctuation.
Comparing most frequent scores given from all of the sites between their most recent scoring and last year, three sites decreased, two sites increased and three sites remained the same. Most of the changes here, or lack thereof, aren’t really that surprising or drastic as the numbers were so close that they can easily swap between two scores that are right next to each other. The largest change was Cigar Dojo going from an 89 to a 91 which is the largest change, but with the low number of reviews and their odd scoring variance this year, it’s not hard to see this happening.
Removed Sites
Cigar Smoke
This site went offline in September of 2019 and has yet to return.
How Bout That Cigar
This site published less than a dozen reviews over the last 12 months. In an article like this where data is key, there just isn’t enough here to be able to continue on with analyzing this site.
Overall Comparison
Comparing all of the sites side by side for their results over the past 12 months provides an interesting look at the majority of review sources that utilize the 100 point system (aside from a few magazines that don’t have scoring available online) and how they stack up against each other in terms of average scores and most frequent scores given.
The sites are ranked from left to right in terms of having lower average scores and lower number for the most frequent score given. The sources that have either given in to inflation or have poor scoring systems are on the right while the sites that have a more consistent or grounded scoring system are on the left.
Final Thoughts
As I opined last year, I again feel that the new releases (which are primarily what is reviewed by most of the sites included in this article) from these past 12 months are a step below in flavor quality than what was released the previous 12 months which were lower than even the 12 months prior to that. Based on that, I find any sites with scoring averages increasing this past year fairly curious.
With the amount of data we now have for all of the sites, it’s fair to make some assumptions. The Cigar Authority and Tiny Tim are just high scorers and it appears that it will just continue. I feel similarly about Stogie Press as the few clunker scores that are thrown in there lower the average, but the frequent scores are telling. Blind Man’s Puff has a decent bell curve, but the scoring system is flawed and leads to the inflated scoring the site exhibits. It’s fixable, but I doubt they do anything. Cigar Dojo has one reviewer that provides a bulk of the reviews and a small contribution from a range of others, so they variance of all of the input create chaos with their numbers. Cigar Coop has a narrow bell curve and it makes the scores for each reviews hard to decipher how different cigars are from each other. I think it’s an important flaw to fix. In regards to Cigar Aficionado, they have the best distribution of scores, and while I typically don’t agree with the scores given to particular cigars, the numbers are appropriate for the system they designed. halfwheel’s scoring this year was a bit interesting due to the gap in the score frequencies, but with them being pretty consistent in previous years, I don’t see it as more than a curiosity at this point.
In using our own data, which doesn’t use the 100 point system, we can show the change in scoring for each reviewer on Developing Palates to give you an indication of what kind of movement has happened. Other than Seth, you can see that the rest of us have all had decreases in average scoring the last two years which goes along with my theory of the flavor quality being down the last couple of years.
With how the cigar industry has functionally changed in regards to factory production and blending this year due to COVID-19 and with it continuing on into 2021, I’ll be interested to see what the quality of flavor will be like in the next 12 months and how it is reflected from these sites.
Leave a Reply