Statistics Levels statistics - Research for monthly reports

@Gepeto Just thinking out loud here, but you could do a "Top 20" and also have a list of the "Top 5 by Up-and-Coming Authors" which could be culled out of the levels ranking 21-50 or something.

 

Let me convert your scoring weights to percentages of what each contributes to the final score.
Multiplier Factors - Rating: 4 / Rates: 3 / Likes: 1 / Views: 2 / Comments: 1
Rating: 4/11 = 36%
Rates: 3/11 = 27%
Likes: 1/11 = 9%
Views: 2/11 = 18%
Comments: 1/11 = 9%
So rating (average stars) and (number of) rates contributes together 63% which is very dominant.

I too think I would prefer a larger list than limiting per author. These designers work hard on these levels so deserve credit for the quality and quantity of work they do. I was surprised at the total # of levels 544 so 30 levels is only the top 5%. Another thought is list all levels achieving over X amounts of points, then you don't have a set number in the list each month but list anything meeting a minimum score.

 

+1

 

To me it seems a little high, I would like more around 50% But as I said before this might be a good thing for a poll once we get some examples to put up.

You are talking about the average rating from here on, so maybe you didn't understand I meant listing anything over X amount of points for the final score. So something like 15 points or more gets on the list. Again maybe another subject of a poll: should we have a set number of levels in our monthly list (and what should that number be), or have a minimum score requirement (and what should that score be).

I am amazed we had 70 5.0 ratings, it would be interesting to see a breakdown of these levels by number of rates.

 

Mmmm, good snacking material! Interesting June is so high. Had people been creating levels with no outlet to share them until then? I think I discovered Mekorama in July.

 

@B Hill's scoring methodology is very interesting, but it seems to me that if we want to find the best levels, comments and views are irrelevant and I wouldn't put them as factors. I also think there are ways to improve it.

I agree that a single 5-star rating says very little about the level (that's probably why we think the number of rates is an important factor). However, when a level has enough rates, say more than 6 rates, I think the rating becomes a more important factor since it's more stablized. For example, if level A has 8 rates and a rating of 5, and level B has 10 rates and a rating of 4.5, then level A would seem like the better one based on its rating.

So we want the rates to have a less influence when it grows larger. I think a good way is to take the rating multiplied by the logarithmic value of rates. With a bit of tuning, we can use

score = rating x log2(rates+2)

If we test it against the example I gave:

score of A = 5log2(8+2) = 16.61
score of B = 4.5log2(10+2) = 16.13

level A wins!

As for likes, I think they are rather unstable and they act more like small bonus points, so I would add 0.2 points for each like. The formula would be:

score = rating x log2(rates+2) + 0.2 x likes

I think this formula has some advantages:
1. It's much more smooth than using quadrants. (It's a continuous function)
2. It's simpler because you don't need to have the ranking for each factor before calculating the score.
3. It makes levels from different months comparable.

 

Ok interesting thought. However what if there were 100 likes (yes that is possible)? Then the likes will add 20 points, becoming dominant in the score. That doesn't seem to be what you would want since you indicated you wanted the rate to be dominant. And it makes cheating easy (tell all your friends to like your level from 5 different devices). You might need to do a log2() on your likes too.

The weights used by Gepeto this time would give the following percentages that each factor contributes to the score.

Rating: 4/10 = 40%
Rates: 3/10 = 30%
Likes: 2/10 = 20%
Views: 1/10 = 10%
Comments: 0/10 = 0%
So rating (average stars) and (number of) rates contributes together 70% which is very dominant.

Using quartiles normalizes the data so that no one factor can be overly dominant above the weighting it is given.

1. I'm not sure what smoothness has to do with it, we just need to calculate a score. It's not like we're ramping up and down RPMs to find the ideal power ratio.
2. True, mathematically it is simpler. But that's why we have computers do the heavy algorithms
3. Using quartiles the levels from different months are also comparable. There is always the same maximum score a level can receive, as long as the same weights are always used.

Richard don't take this as harsh criticism, I think you have a very respectable idea, with some tweaking to prevent one factor from overtaking the whole score. As I have said before, this is a good thing to flesh out and possibly take final ideas to a poll. So now we have two options to consider:
Quartile scoring or a formula like score=rating X log2(rates + 2) + log2(likes + 2) + log2(comments + 2)

 

Also let me just point out that comparing the levels in the lists, there are only a difference of 6 levels. I am not comparing actual rank at all since they are all good and am just considering which ones made it on the lists.

On Quartiles list but not Log2:
21- Bot on Ball Demo
26- The Mountain
27- Electric Cute
28- A lofty Goal
29- Swivel
30- Ancient Temple

On Log2 but not on quartiles:
21- Ship Picking
22- Super Mario Bro v3
23- Sudoku
24- Into The Hole
25- Mini Tilt
30- Face Evil Inside

I attribute this difference to the fact that levels on the quartile list numbered 26 to 30 got 3 or less rates, so using the log2 formula that seems to be the cut-off that a level must get at least 4 rates to make it onto the log 2 list. Bot-on-Ball Demo had 0 likes so that may have prevented it from doing well with the log2 formula.

 

I know I don't use likes, if I enjoy a level I will comment and rate it. So I'm not really a fan of using likes to contribute much to the score. Views is also problematic since even guests are counted when they view a level, but guests cannot make comments or ratings.

 

@B Hill Thanks for the feedback! I have thought of it and that's why I made the weight as small as 0.2. The most likes any level has received in this forum is about 20 and that means 4 extra points, which I think is decent. Like @Frenzies, I don't think 100 likes is going to happen and this kind of cheating is easy to spot. However taking the log value of it could make it safer, so the formula can be modified to be:
score = rating x log2(rates+2) + log2(likes+2)

So log2(likes+2) always gives at least one point even when there are no likes, but I think there's no problem with that.

1. I looked through the thread and I haven't found how you or @Gepeto divided the quartiles exactly, but I'll take @Gepeto's 20 most rated chart at the beginning for example. Assuming you divide it evenly into 4 quartiles, levels with 10 to 13 rates all go to Q1, which gives them a score of 4. That means the methodology does not distinguish levels with 10 and 13 rates when all other factors are the same. The same could happen for ratings and likes. That's what I mean by the smoothness problem. Please tell me if I misunderstood anything.
2. There's no harm with making it simpler
3. The quartiles come from the rankings of each month, and that means the score of a level depends on the performance of other levels in that month. If the level is moved to a different month, it may have a different ranking and thus a different score. This wouldn't happen with my formula.

 

Looking at the quartiles list, I think the top half of it performs quite well. In the lower part though, there are some levels with 3 rates and a rating of 4.0, and there's a level with only 1 rate. Obviously some levels outside of this list have better statistics and should be in their places.

 

What about using log2(likes+1), that way zero likes gives 0 towards the score. I was going to ask, but forgot, why you picked +2?

1. Those lists are the end result of using quartiles on ALL levels from the last month. But you get the basic idea. We aren't really trying to determine, which level is #1, which is #2, etc in a strict why to find the absolute best level. We are in a more general way finding the best collection of levels. Yes Gepeto is ranking them 1 to 20, but that's not really the primary intent, we just want a collection of the best levels. Double point precision (or smoothness) isn't really necessary.
2. True simple or not doesn't really matter, it is what is the end result that matters.
3. True, the score does depend on the performance of other levels in that month, and that is exactly what we are trying to determine. I think you are thinking about comparing levels across different months by their score, and that really wasn't the intent. We were thinking of creating a list at the end of each month for a guide for new members to be able to find great levels. We aren't really trying to compare levels month to month.
Also even with your scoring method, as time goes on and people go back and try older levels and rate them, the score will change since the rates, rating and likes values will change with time. If you try to compare levels across months, what score would you use, the original score calculated at the end of the month, or the score using up to date values for rates , etc?

But I think this is a great discussion to have and the formula idea is a great option to consider.

Also Gepeto now lists 30 levels which seems to be a good inclusive number but that can also be discussed (or polled).