1. Announcing Mekorama on the Web!

    Now anyone can play levels from the forum online, with one click!

    Dismiss Notice
  2. Psst! If you're new here, welcome! Please visit these pages first for information about the forum and Mekorama:

    Welcome! ¡Bienvenido! Selamat datang! Добро пожаловать! Willkommen!
    and
    Everything you want to know about Mekorama

    Dismiss Notice

Statistics Levels statistics - Research for monthly reports

Discussion in 'Notable Levels and Other Cards' started by Gepeto, Oct 6, 2016.

  1. sawdust

    sawdust Retired Moderator

    Messages:
    149
    Levels:
    83
    Albums:
    4
    Likes Received:
    771
    Joined:
    Jun 8, 2016
    @Gepeto Just thinking out loud here, but you could do a "Top 20" and also have a list of the "Top 5 by Up-and-Coming Authors" which could be culled out of the levels ranking 21-50 or something.
     
  2. B Hill

    B Hill Active Member

    Messages:
    63
    Levels:
    24
    Albums:
    3
    Likes Received:
    251
    Joined:
    Jul 28, 2016
    Let me convert your scoring weights to percentages of what each contributes to the final score.
    Multiplier Factors - Rating: 4 / Rates: 3 / Likes: 1 / Views: 2 / Comments: 1
    Rating: 4/11 = 36%
    Rates: 3/11 = 27%
    Likes: 1/11 = 9%
    Views: 2/11 = 18%
    Comments: 1/11 = 9%
    So rating (average stars) and (number of) rates contributes together 63% which is very dominant.

    I too think I would prefer a larger list than limiting per author. These designers work hard on these levels so deserve credit for the quality and quantity of work they do. I was surprised at the total # of levels 544 so 30 levels is only the top 5%. Another thought is list all levels achieving over X amounts of points, then you don't have a set number in the list each month but list anything meeting a minimum score.
     
  3. sawdust

    sawdust Retired Moderator

    Messages:
    149
    Levels:
    83
    Albums:
    4
    Likes Received:
    771
    Joined:
    Jun 8, 2016
    +1
     
  4. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    It would be great but I don't think it is in the scope of the algorithm as it needs a review not only based on the levels but on the involvment of an author. Something else based on the increasing post and evaluations of a creator maybe could do that but it would probably need an access to the database in order to "track" the progress of someone. From what I do, I am focusing on the levels and try to build a relevant "push button" for monthly report.

    So? Does it sounds good to you? Because to me it seems fine as views are not underestimated but not too high as the other parameters.

    Yes, but that should be sufficient to have the "crème de la crème" :D I suppose I will switch to 30 levels.

    That's a great idea. According to it I have quickly made a test order by [Rating > Rates > Likes > Views > Comments] with (rating > 4.00) and I have found... 120 levels stating with 4.33 :eek:

    I am doing on live so here is more detailed results

    Rating > ...Matchminimum
    4.301204.33
    4.60844.67
    4.70764.75
    4.80714.82
    4.90705.00
    I have been surprised, so I have made another research based on : How many occurences per rating can be found?

    RatingMatch
    0.00299
    1.002
    2.004
    3.0029
    3.331
    3.401
    3.5014
    3.673
    3.751
    4.0070
    4.3313
    4.5016
    4.678
    4.607
    4.753
    4.802
    4.821
    5.0070
    And that's only for september... So basically even ratings with 5.00 would lead to a huge table of 70 results :confused: Of course these are results with even 1 rate so I'll try some other combinaisons with more than 1 or 2.
     
  5. B Hill

    B Hill Active Member

    Messages:
    63
    Levels:
    24
    Albums:
    3
    Likes Received:
    251
    Joined:
    Jul 28, 2016
    To me it seems a little high, I would like more around 50% But as I said before this might be a good thing for a poll once we get some examples to put up.

    You are talking about the average rating from here on, so maybe you didn't understand I meant listing anything over X amount of points for the final score. So something like 15 points or more gets on the list. Again maybe another subject of a poll: should we have a set number of levels in our monthly list (and what should that number be), or have a minimum score requirement (and what should that score be).

    I am amazed we had 70 5.0 ratings, it would be interesting to see a breakdown of these levels by number of rates.
     
  6. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    Haha :) Yes. By points I have understood "rating" o_O
    Indeed I have found this interesting. So to end up on that here are some more results

    RatingRates > 1Rates > 2Rates > 3
    5.0362720
    4.82111
    4.8222
    4.75333
    4.6788-
    4.6777
    4.51622
    4.3313131
    4.02773
    3.75111
    3.6733-
    3.514--
    3.4111
    3.3311-
    3.0103-
    I suppose releasing a TOP 30 would be sufficient as for september score > 15 is ranked until #24 in the list.
     
  7. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    As we're speaking about statistics I wondered how the uploads per day evolved since june.
    Here are some diagrams to show it month after month, day after day. Maybe it could be interesting to join it to monthly TOP 30 but it doesn't really help about the best levels of the month. Just stats...

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    I'm not doing much more stats arround that. Just curiousity I want to share :)
     
  8. Frenzies

    Frenzies Administrator Staff Member

    Messages:
    355
    Levels:
    18
    Albums:
    5
    Likes Received:
    1,215
    Joined:
    Jun 15, 2016
    That last day in August was when @S. Nagy uploaded his album... just a heads up to those who don't know and are wondering why it was so high.
     
    Gepeto likes this.
  9. B Hill

    B Hill Active Member

    Messages:
    63
    Levels:
    24
    Albums:
    3
    Likes Received:
    251
    Joined:
    Jul 28, 2016
    Mmmm, good snacking material! Interesting June is so high. Had people been creating levels with no outlet to share them until then? I think I discovered Mekorama in July.
     
  10. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    @B Hill - I thought the same thing: I expected the uploads to increase by the time. According to Mekorama's official website the game has been released on May 15, 2016. So not so long time between the release and the forum (with the time to really discover the in-game levels?)
     
    Last edited: Oct 13, 2016
  11. richardfu

    richardfu Moderator

    Messages:
    99
    Levels:
    50
    Albums:
    2
    Likes Received:
    677
    Joined:
    May 23, 2016
    @B Hill's scoring methodology is very interesting, but it seems to me that if we want to find the best levels, comments and views are irrelevant and I wouldn't put them as factors. I also think there are ways to improve it.

    I agree that a single 5-star rating says very little about the level (that's probably why we think the number of rates is an important factor). However, when a level has enough rates, say more than 6 rates, I think the rating becomes a more important factor since it's more stablized. For example, if level A has 8 rates and a rating of 5, and level B has 10 rates and a rating of 4.5, then level A would seem like the better one based on its rating.

    So we want the rates to have a less influence when it grows larger. I think a good way is to take the rating multiplied by the logarithmic value of rates. With a bit of tuning, we can use

    score = rating x log2(rates+2)

    If we test it against the example I gave:

    score of A = 5log2(8+2) = 16.61
    score of B = 4.5log2(10+2) = 16.13

    level A wins!

    As for likes, I think they are rather unstable and they act more like small bonus points, so I would add 0.2 points for each like. The formula would be:

    score = rating x log2(rates+2) + 0.2 x likes

    I think this formula has some advantages:
    1. It's much more smooth than using quadrants. (It's a continuous function)
    2. It's simpler because you don't need to have the ranking for each factor before calculating the score.
    3. It makes levels from different months comparable.
     
    Last edited: Oct 22, 2016
    meko, TR O, Frenzies and 3 others like this.
  12. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    That's a good point, I like this idea.

    On the next post here is the result for Quartiles and Richardfu methodology for comparison. It seems that a level has been deleted by the time... I have disabled comments from the Quartiles method but kept a low weight for the views. I believe views represent something while they are combine with other criteria: the popularity of a level not based on membership only.

    Both are interesting...
     
    Last edited: Oct 23, 2016
  13. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    Stats from the 09/01/2016 to the 09/30/2016 included (MM/DD/YYYY) - Made on 10/22/2016
    Number of levels: 543

    Quartiles statistical Methodology - Rating: 4 / Rates: 3 / Likes: 2 / Views: 1 / Comments: 0

    #TitleAuthorViewsRatesRatingLikesCommentsScoreDate
    1Limborichardfu_666115.05154009/02/2016
    2Path of Couragerichardfu_399105.03133709/30/2016
    3Expedition IIIrichardfu_37695.0393609/06/2016
    4PetroleumGepeto55685.03173609/01/2016
    5Opening Doorsmeko66975.04223509/12/2016
    6Mountain Depthrichardfu_40675.05103409/10/2016
    7Square one IIvince43355.02173209/21/2016
    8Crouching TigerTR O36385.04173109/27/2016
    9Chateau Sawdustsawdust25585.0483109/23/2016
    10Fishingrichardfu_28465.0373009/18/2016
    11Expedition IVrichardfu_440105.01163009/12/2016
    12The Twins IIcpw19155.0082809/25/2016
    13JourneyKhudrat23355.0292809/23/2016
    14Glyph Walkrichardfu_534114.825192309/16/2016
    15No way to WinTR O53654.63292309/12/2016
    166 Piece Burrsawdust18945.0472209/28/2016
    17Expedition Vrichardfu_31065.0272209/22/2016
    18Cae11Chuckthulhu23454.8392009/19/2016
    19Screenshot_2016-09-08-23-01-41-47Muslim Arizzy20454.8342009/08/2016
    20Special Forces Vol IS. Nagy52145.01191809/04/2016
    21Bot-on-Ball Demoretrograde18245.0071609/26/2016
    22Slider Jump Demoretrograde18645.0171609/25/2016
    23Quidditchmeko27945.0181609/25/2016
    24Back and Forth 1.1cpw22145.0171609/11/2016
    25Mechanical Poolrichardfu_19654.8141209/10/2016
    26The Mountainmeko62834.04211009/09/2016
    27Electric CuteLooKing?15634.036809/30/2016
    28A lofty goalJose Luis Galiano17634.3337809/30/2016
    29SwivelMuslim Arizzy20934.67313809/13/2016
    30Ancient TempleLazarus17415.051809/04/2016

    Richardfu methodology - score = rating x log2(rates+2) + 0.2 x likes (comments and views aren't considered)

    #TitleAuthorViewsRatesRatingLikesCommentsScoreDate
    1Limborichardfu_666115.051519.5009/02/2016
    2Glyph Walkrichardfu_534114.8251918.8409/16/2016
    3Path of Couragerichardfu_399105.031318.5209/30/2016
    4Expedition IVrichardfu_440105.011618.1209/12/2016
    5Expedition IIIrichardfu_37695.03917.9009/06/2016
    6Crouching TigerTR O36385.041717.4109/27/2016
    7Chateau Sawdustsawdust25585.04817.4109/23/2016
    8PetroleumGepeto55685.031717.2109/01/2016
    9Mountain Depthrichardfu_40675.051016.8509/10/2016
    10Opening Doorsmeko66975.042216.6509/12/2016
    11Fishingrichardfu_28465.03715.6009/18/2016
    12Expedition Vrichardfu_31065.02715.4009/22/2016
    13JourneyKhudrat23355.02914.4409/23/2016
    14Square one IIvince43355.021714.4409/21/2016
    15Cae11Chuckthulhu23454.83914.0809/19/2016
    16Screenshot_2016-09-08-23-01-41-47Muslim Arizzy20454.83414.0809/08/2016
    17The Twins IIcpw19155.00814.0409/25/2016
    186 Piece Burrsawdust18945.04713.7209/28/2016
    19Mechanical Poolrichardfu_19654.81413.6809/10/2016
    20No way to WinTR O53654.632913.5109/12/2016
    21Ship Pickingrichardfu_19464.332513.3909/27/2016
    22Super Mario Bros V3sawdust38154.621813.3109/30/2016
    23SudokuMScript25054.621213.3109/28/2016
    24Into The HoleKhudrat15454.62513.3109/22/2016
    25Mini Tiltmeko29654.621313.3109/16/2016
    26Slider Jump Demoretrograde18645.01713.1209/25/2016
    27Quidditchmeko27945.01813.1209/25/2016
    28Back and Forth 1.1cpw22145.01713.1209/11/2016
    29Special Forces Vol IS. Nagy52145.011913.1209/04/2016
    30FACE EVIL INSIDE (RE-EDIT & BARCODE)Widha Wowok29554.611113.1109/17/2016
     
    Last edited: Oct 23, 2016
    TR O and cpw like this.
  14. B Hill

    B Hill Active Member

    Messages:
    63
    Levels:
    24
    Albums:
    3
    Likes Received:
    251
    Joined:
    Jul 28, 2016
    Ok interesting thought. However what if there were 100 likes (yes that is possible)? Then the likes will add 20 points, becoming dominant in the score. That doesn't seem to be what you would want since you indicated you wanted the rate to be dominant. And it makes cheating easy (tell all your friends to like your level from 5 different devices). You might need to do a log2() on your likes too.

    The weights used by Gepeto this time would give the following percentages that each factor contributes to the score.

    Rating: 4/10 = 40%
    Rates: 3/10 = 30%
    Likes: 2/10 = 20%
    Views: 1/10 = 10%
    Comments: 0/10 = 0%
    So rating (average stars) and (number of) rates contributes together 70% which is very dominant.

    Using quartiles normalizes the data so that no one factor can be overly dominant above the weighting it is given.

    1. I'm not sure what smoothness has to do with it, we just need to calculate a score. It's not like we're ramping up and down RPMs to find the ideal power ratio.
    2. True, mathematically it is simpler. But that's why we have computers do the heavy algorithms :)
    3. Using quartiles the levels from different months are also comparable. There is always the same maximum score a level can receive, as long as the same weights are always used.

    Richard don't take this as harsh criticism, I think you have a very respectable idea, with some tweaking to prevent one factor from overtaking the whole score. As I have said before, this is a good thing to flesh out and possibly take final ideas to a poll. So now we have two options to consider:
    Quartile scoring or a formula like score=rating X log2(rates + 2) + log2(likes + 2) + log2(comments + 2)
     
    TR O likes this.
  15. B Hill

    B Hill Active Member

    Messages:
    63
    Levels:
    24
    Albums:
    3
    Likes Received:
    251
    Joined:
    Jul 28, 2016
    Also let me just point out that comparing the levels in the lists, there are only a difference of 6 levels. I am not comparing actual rank at all since they are all good and am just considering which ones made it on the lists.

    On Quartiles list but not Log2:
    21- Bot on Ball Demo
    26- The Mountain
    27- Electric Cute
    28- A lofty Goal
    29- Swivel
    30- Ancient Temple

    On Log2 but not on quartiles:
    21- Ship Picking
    22- Super Mario Bro v3
    23- Sudoku
    24- Into The Hole
    25- Mini Tilt
    30- Face Evil Inside

    I attribute this difference to the fact that levels on the quartile list numbered 26 to 30 got 3 or less rates, so using the log2 formula that seems to be the cut-off that a level must get at least 4 rates to make it onto the log 2 list. Bot-on-Ball Demo had 0 likes so that may have prevented it from doing well with the log2 formula.
     
  16. Frenzies

    Frenzies Administrator Staff Member

    Messages:
    355
    Levels:
    18
    Albums:
    5
    Likes Received:
    1,215
    Joined:
    Jun 15, 2016
    @B Hill I just read the top part of your post, but hopefully the bottom part just explains the formula.

    Not much people like levels. The highest is 26, I think, and that's @richardfu's Return Trip, which also has one (if not the most) of the highest views and ratings. Since you need an account to rate, I don't think that asking your friends to do it would affect it that much. And even if you do, they're likely to ignore you (like my friends whenever I talk to them about Mekorama), and I won't blame them. Why would they create an account just to "like" one of your levels? Besides, the community here seems nice and I don't think they would try to cheat like that.
     
    Last edited: Oct 23, 2016
  17. B Hill

    B Hill Active Member

    Messages:
    63
    Levels:
    24
    Albums:
    3
    Likes Received:
    251
    Joined:
    Jul 28, 2016
    I know I don't use likes, if I enjoy a level I will comment and rate it. So I'm not really a fan of using likes to contribute much to the score. Views is also problematic since even guests are counted when they view a level, but guests cannot make comments or ratings.
     
    Frenzies likes this.
  18. richardfu

    richardfu Moderator

    Messages:
    99
    Levels:
    50
    Albums:
    2
    Likes Received:
    677
    Joined:
    May 23, 2016
    @B Hill Thanks for the feedback! I have thought of it and that's why I made the weight as small as 0.2. The most likes any level has received in this forum is about 20 and that means 4 extra points, which I think is decent. Like @Frenzies, I don't think 100 likes is going to happen and this kind of cheating is easy to spot. However taking the log value of it could make it safer, so the formula can be modified to be:
    score = rating x log2(rates+2) + log2(likes+2)

    So log2(likes+2) always gives at least one point even when there are no likes, but I think there's no problem with that.

    1. I looked through the thread and I haven't found how you or @Gepeto divided the quartiles exactly, but I'll take @Gepeto's 20 most rated chart at the beginning for example. Assuming you divide it evenly into 4 quartiles, levels with 10 to 13 rates all go to Q1, which gives them a score of 4. That means the methodology does not distinguish levels with 10 and 13 rates when all other factors are the same. The same could happen for ratings and likes. That's what I mean by the smoothness problem. Please tell me if I misunderstood anything.
    2. There's no harm with making it simpler:)
    3. The quartiles come from the rankings of each month, and that means the score of a level depends on the performance of other levels in that month. If the level is moved to a different month, it may have a different ranking and thus a different score. This wouldn't happen with my formula.
     
  19. richardfu

    richardfu Moderator

    Messages:
    99
    Levels:
    50
    Albums:
    2
    Likes Received:
    677
    Joined:
    May 23, 2016
    Looking at the quartiles list, I think the top half of it performs quite well. In the lower part though, there are some levels with 3 rates and a rating of 4.0, and there's a level with only 1 rate. Obviously some levels outside of this list have better statistics and should be in their places.
     
  20. B Hill

    B Hill Active Member

    Messages:
    63
    Levels:
    24
    Albums:
    3
    Likes Received:
    251
    Joined:
    Jul 28, 2016
    What about using log2(likes+1), that way zero likes gives 0 towards the score. I was going to ask, but forgot, why you picked +2?

    1. Those lists are the end result of using quartiles on ALL levels from the last month. But you get the basic idea. We aren't really trying to determine, which level is #1, which is #2, etc in a strict why to find the absolute best level. We are in a more general way finding the best collection of levels. Yes Gepeto is ranking them 1 to 20, but that's not really the primary intent, we just want a collection of the best levels. Double point precision (or smoothness) isn't really necessary.
    2. True simple or not doesn't really matter, it is what is the end result that matters.
    3. True, the score does depend on the performance of other levels in that month, and that is exactly what we are trying to determine. I think you are thinking about comparing levels across different months by their score, and that really wasn't the intent. We were thinking of creating a list at the end of each month for a guide for new members to be able to find great levels. We aren't really trying to compare levels month to month.
    Also even with your scoring method, as time goes on and people go back and try older levels and rate them, the score will change since the rates, rating and likes values will change with time. If you try to compare levels across months, what score would you use, the original score calculated at the end of the month, or the score using up to date values for rates , etc?

    But I think this is a great discussion to have and the formula idea is a great option to consider.

    Also Gepeto now lists 30 levels which seems to be a good inclusive number but that can also be discussed (or polled).
     
    richardfu and TR O like this.

Share This Page