1. Announcing Mekorama on the Web!

    Now anyone can play levels from the forum online, with one click!

    Dismiss Notice
  2. Psst! If you're new here, welcome! Please visit these pages first for information about the forum and Mekorama:

    Welcome! ¡Bienvenido! Selamat datang! Добро пожаловать! Willkommen!
    and
    Everything you want to know about Mekorama

    Dismiss Notice

Organization How can we save the rating system?

Discussion in 'General (Issues, Help, Discussions)' started by Chuckthulhu, Jan 18, 2017.

Tags:
  1. Chuckthulhu

    Chuckthulhu Spanish Moderator

    Messages:
    76
    Levels:
    158
    Albums:
    10
    Likes Received:
    2,304
    Joined:
    Sep 5, 2016
    So I've been reading threads, chatting in Discord and so on about this topic and I finally decided about posting it for discussion in the forum.

    I love this game and this community, I spend hours in Mekorama related activities, I have created about 85 levels and 7 remakes of some of those levels and I have no idea how many levels I played but it's easily a couple hundreds... and I have noticed that levels are rated 5 stars too often to the point that it really is hard to determine which levels are better than others based on ratings.

    Don't get me wrong! , I love receiving notifications about my levels getting 5 star ratings, but I think leaving a 3 and 4 star rating is fine!

    The way I see it, receiving comments it's better than not getting anything, even "deserved" 1 star ratings may be better; as a creator it's great to know what you did right and what you did wrong, especially when you're just starting.

    This community being what it is, I feel that we can all agree that we don't want to hurt the creators, most of us are creators sharing ideas and appreciation for the other's work, so how come we all think all levels deserve 5 stars?

    I think anything from 3 to 5 stars, if deserved and properly explained is a good rating, and 1 or 2 stars ratings, although "bad", can help to receive feedback about what we can improve. As long as we remember that we want to encourage creators to continue creating and making better levels, we all win. What do you guys think?
     
    QuantumForce, vince, cpw and 3 others like this.
  2. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    I agree with you about the 3 to 5 stars. I don't remember giving 1 or 2. If someone wants to give a smart advice about something bad he can do it throught comments, not rating. Giving 1 or 2 sounds to me just like "you did bad and here is your punishment for that". If I don't like a level I just ignore or leave a comment if I feel something interesting wanted to be done but failed.

    As I already said I use much more the stars as like/love/adore (so to express my feelings) than to "evaluate" a level, despite it stills an evaluation. Comments can give more nuances and details on what's wrong or great and are much more important to communicate in a good way.

    Everybody learn from errors and mistakes so that's important to have positives and negatives feedbacks as far as they are for the better and as we stay opened to constructive criticism.
     
    hadi, QuantumForce, vince and 2 others like this.
  3. vince

    vince Famous Member

    Messages:
    67
    Levels:
    44
    Albums:
    1
    Likes Received:
    957
    Joined:
    Jun 27, 2016
    I agree with @Chuckthulhu on the fact that 5 stars ratings may be given too easily (by me included). However, I can think of a couple of reasons for that:
    The number of ratings are generally quite low, which makes the average value a bit unreliable to evaluate the quality of a level, because only one bad rating can greatly decrease the average. It is not like if we had thousands of ratings per level. That's why I rely more on the number of ratings than on the average value. To answer to the question, I think that the rating system loses its value because of the little number of ratings.
    Another reason is that I feel awkward judging and giving a grade to a level like if I was some kind of teacher. If a level is not perfect from my point of view, I will tell it in the comment, and maybe give 5 stars anyway because the level was enough interesting for me to take the time to comment and give my advice for improvement. If it's really bad or not my cup of tea, I won't rate at all.
    I know it is not the most effective behaviour to evaluate levels, but again, the number of ratings is too low to rely on a few subjective number of stars.
     
    Last edited: Jan 18, 2017
    Chuckthulhu and Gepeto like this.
  4. Chuckthulhu

    Chuckthulhu Spanish Moderator

    Messages:
    76
    Levels:
    158
    Albums:
    10
    Likes Received:
    2,304
    Joined:
    Sep 5, 2016
    I like the way you see the rating system in such a positive way @Gepeto and you make an excellent point about the number of ratings @vince!

    Something else we talked about was that if the rating system was from 1 to 10, there would probably be more 8-9 ratings and players would probably reserve the 10 stars rating for those exceptional levels, this is also the reason why I think the 5 stars rating has lost its value, we have had several levels rated with only 5 stars because maybe we thinking that rating anything below that could be offensive/hurtful for the creator, but a 4 star rating it's still good to me.

    The way I see it, if I had one 5stars rating and one 4stars rating, my average would be 4.5, that's still a good rating! The problem would be if I had tons of 1-3 stars ratings decreasing the average.

    A good practice I've been adopting is that, if I see some 3-4 stars ratings and I think the level is something like a 4.5, I'll leave a 5 and make the comment about it, which doesn't mean I want everybody to rate based on other's ratings, but I think it's something we can do taking advantage that this is a community. Maybe we can increase the number of ratings and interactions in the levels if we agree that using 3 and 4 stars as a rating it's not a bad thing... But if 5 stars is the lowest we will ever rate, anyone can feel bad when receiving a 4!
     
    Gepeto and vince like this.
  5. vince

    vince Famous Member

    Messages:
    67
    Levels:
    44
    Albums:
    1
    Likes Received:
    957
    Joined:
    Jun 27, 2016
    Actually I think that a rating system from 0 to 10 could be better. Receiving a 9 feels a lot better than receiving a 4 (even if it is almost the same thing), so it could be more widely used by everyone, even by new members who don't necessary know about a "4 stars is quite good" agreement. But I don't think it is possible to change now the rating range to 10...
     
    Chuckthulhu likes this.
  6. Chuckthulhu

    Chuckthulhu Spanish Moderator

    Messages:
    76
    Levels:
    158
    Albums:
    10
    Likes Received:
    2,304
    Joined:
    Sep 5, 2016
    Yes, the more I think about it, the more I like the idea of 1-10 stars rating. As it is, we would rarely rate under 6 (3 in the current system) and although it's still almost the same, 6-10 feels like more options to "like-dislike" something in the level than 3-5.
    I also like to think that maybe a player didn't like a level because he likes puzzles over ability? Maybe a player doesn't like levels with exploits compared with pure logic? There's nothing wrong in saying that so other players that have similar taste will go by your review, an example "Chuck likes easy puzzles that can be solved in minutes, X likes long levels with story and obstacles that are more challenging, for this level Chuck rated 5 and X rated 4, so if I like similar levels to Chuck I think I'm going to like it and rate it a 5 as well...".
    And as an alternative, maybe not 1-10 but even being able to rate x.5 would be nice... maybe you feel like rating 4.5 or 3.5...
    I don't know if the rating system can be changed though... :rolleyes:o_O
     
    Frenzies and vince like this.
  7. Fish

    Fish Member

    Messages:
    1
    Levels:
    19
    Albums:
    2
    Likes Received:
    30
    Joined:
    Jan 23, 2017
    How about a system of ratings for different characteristics of the level. i.e. Creativity/Originality, Difficulty, Length, etc. ?
     
    Chuckthulhu likes this.
  8. Chuckthulhu

    Chuckthulhu Spanish Moderator

    Messages:
    76
    Levels:
    158
    Albums:
    10
    Likes Received:
    2,304
    Joined:
    Sep 5, 2016
    That would be awesome and I think we should all consider those factors when rating, but it could also be counter-productive because some users would feel it's too much work to rate it now.
    As @vince pointed out, a factor that hurts the rating system is also the number of ratings, so I think what's better is to encourage more users to give ratings without feeling that it's offensive to rate below 5 stars, even if you feel it's a 4 or a 3 and ideally have solved it, considered the factors you mentioned and can leave a feedback backing up your rating that could also work as constructive feedback for the creator...
    But the more I think about it, the more I'd like the rating system to change to 10 stars for that reason, so you could rate with 8 and 9 and feel like you still value the creator's work, but maybe now you reserve that 10 for what you consider exceptional levels!
     
    vince likes this.
  9. QuantumForce

    QuantumForce Active Member

    Messages:
    37
    Levels:
    3
    Albums:
    1
    Likes Received:
    113
    Joined:
    Oct 19, 2016
    Is it even possible to change the forum's rating system? Do the mods have access to forum plugins or whatever?
     
    Chuckthulhu likes this.
  10. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    @nGord have an access to the admin panel of the forum but this is not covering low level works. It is much more about some priviledge permissions (giving someone moderator right, as an example), managing warning, bans and forum messages, nodes, logs, users among other stuff. To change or add a plugins someone needs to be super-admin (currently only NeoCHI), which requires to have an access to the filesystem of the server and modify some files.

    If possible, I'm not even sure we can change the rating system without loosing the previous one :eek:. That really seems dangerous anyway. But talking on a more efficient rating system still interesting... despite probably not applicable. :(
     
    QuantumForce and Chuckthulhu like this.
  11. QuantumForce

    QuantumForce Active Member

    Messages:
    37
    Levels:
    3
    Albums:
    1
    Likes Received:
    113
    Joined:
    Oct 19, 2016
    It sounds more practical to just add a site message with the rating scale. Something like:
    1. Garbage level (lacks goal, obviously incomplete, etc)
    2. Valid level, but poorly designed and not fun
    3. Acceptable level, but some problems or unenjoyable elements
    4. Great level, excellent gameplay and interesting mechanics
    5. Outstanding level. Exemplary gameplay, aesthetics, mechanisms, etc. Should be featured on facebook.
     
    Gepeto and Chuckthulhu like this.
  12. Chuckthulhu

    Chuckthulhu Spanish Moderator

    Messages:
    76
    Levels:
    158
    Albums:
    10
    Likes Received:
    2,304
    Joined:
    Sep 5, 2016
    I like this,

    I think a 3 is more like Good/Acceptable but I do like the idea of making the 4 a "Great level" and keeping the 5s for "Outstanding", I keep thinking, if we were to rate all 50 levels with the scale from 1 to 5, would all of them be 5? probably not... we always have favorite levels and there are some that are clearly just to introduce a feature so they feel like a demo. Following that thought, if there were tons of 3s and 4s in the original levels and we really saved the 5s for our favorite levels in the originals, how come we're giving them away for all levels in the forum?

    I know we don't want to discourage creators, but if we agree that a 3 is "good" and a 4 is "great" I'll take those ratings as a way to improve my next levels instead of unreasonable criticism and, if we take the ratings as a matter of personal preference as well, I'll take a user's 4 on my "ability level" as a way of saying "I prefer logic/puzzle levels, but I still liked your level" instead of "your level is a failure!".

    And again, I think we should encourage more users to rate honestly and comment on the level and the rating instead of being afraid of discouraging creators for rating anything lower than 5... the way it's being used right now, a 5 is almost as indicative of a level's quality as a like!
     
    Gepeto likes this.
  13. Gepeto

    Gepeto MekoStudio Architect Staff Member

    Messages:
    453
    Levels:
    48
    Albums:
    1
    Likes Received:
    2,504
    Joined:
    Jul 7, 2016
    I like it too.

    Maybe someyhing is missing: zero star. :)
    Despite that doesn't express something specific but avoid to be too much critical. Then, we could have next to @QuantumForce suggestions :
    1. Valid level, but poorly designed and not fun
    2. Acceptable level, but some problems or unenjoyable elements
    3. Good level (but can be improved?)
    4. Great level, excellent gameplay and interesting mechanics
    5. Outstanding level. Exemplary gameplay, aesthetics, mechanisms, etc. Should be featured on facebook
    I really like the outstanding point of view for #5 and the idea that it should be featured on facebook!
     
    Chuckthulhu likes this.
  14. nGord

    nGord Standby Administrator, Retired Moderator Staff Member

    Messages:
    608
    Levels:
    7
    Albums:
    4
    Likes Received:
    2,827
    Joined:
    Jul 21, 2016
    I've talked many times about this subject and failed to comment here before because I had already repeated myself so many times elsewhere (profile posts, Discord, level comments). That being said, I thought it time to bump this conversation to the top again. So here are some of my thoughts (again):

    Although I agree with @vince about there being too few ratings to really do justice to evaluating a level, I disagree with his approach of 5-stars or nothing. I say just leave a like if you don't want to use the rating system. You either "like" the level or you don't.

    Unlike @Gepeto, I don't think 1 or 2 stars are a punishment. I believe that their use are an honest evaluation and proper use of the rating system. Again, if the position is to not use all of the ratings available, why use them at all?

    Regarding the expansion of the ratings to 10 stars, this is a moot point because there is no way to change it even if we wanted to.

    So that leaves me in complete agreement with @Chuckthulhu on pretty much everything, particularly his opening remarks and especially about this comment:
    So this brings us back again to what should each of the ratings represent and can we all agree to use such a rating? As much as I also like @QuantumForce's summary, it may be hard to encourage new users of the forum to use anything too different from the base words associated with each of the star levels already in the software (i.e. the words that pop up for each of the stars). These are: 1-star: terrible, 2-star: poor, 3-star: average, 4-star: good, 5-star: excellent.

    Since most of the membership is the remaining absolutely devoted fans of the game, I would argue that the average level uploaded is a good level. This brings me to my biggest point: 3-stars is a truly acceptable rating that signifies that the level is a proper and enjoyable level, a good level that's also the average of what a player is accustomed to and tends to rate. This further means that 4-stars is above average and 5-stars is truly exceptional (far from the average).

    TL;DR - My position is that if one does not want to use the rating system ethically, consistently, and fully, then they should avoid it altogether. Either way, the best thing is to leave a detailed, constructive, and thoughtful comment. Simply leaving a smiley, writing something short like "nice level," or using only terms like "interesting" don't add any value except serving the ego of the commenter. Clicking "like" for such comments promotes poor behaviour and standards.

    //end rant (sorry :()
     
    Last edited: Sep 27, 2017
  15. Bammeras

    Bammeras New Member

    Messages:
    4
    Likes Received:
    5
    Joined:
    Jan 4, 2018
    Oh , maybe it can help us ,
     
  16. nGord

    nGord Standby Administrator, Retired Moderator Staff Member

    Messages:
    608
    Levels:
    7
    Albums:
    4
    Likes Received:
    2,827
    Joined:
    Jul 21, 2016
    Unfortunately, I see three problems with the site message route:

    1. It will most undoubtedly lead to arguments,

    2. It will remove the freedom of making a more subjective assessment and thus many will feel boxed in and in turn will spend less effort on the comments to go along with their ratings, and

    3. Setting a system now will confuse people looking at the ratings from before as they will all be skewed in comparison.
     
  17. philandy

    philandy New Member

    Messages:
    3
    Levels:
    4
    Albums:
    1
    Likes Received:
    5
    Joined:
    Jun 27, 2018
    Thanks for a rating guide! Tips on leaving a better review?

    What about a checkbox for a map designed to simply showcase a mechanic?

    What about vetting, where a community mod beats the level so it gets sort of a seal of approval?
     
  18. nGord

    nGord Standby Administrator, Retired Moderator Staff Member

    Messages:
    608
    Levels:
    7
    Albums:
    4
    Likes Received:
    2,827
    Joined:
    Jul 21, 2016
    Those are not to be shared in the Levels section. Instead they should be shared in this thread: Sharing new (and old) mechanisms or (depending on the mechanic) in other threads within either the Game Behaviours or Level Creation Help sub-forums.

    I think this would solve a problem that we don't have. The majority of levels uploaded are legit. It is also easier to crowdsource the opinions of the players to determine such a seal of approval. That is one way in how @richardfu_ narrows down which levels he should consider to feature on Facebook.

    Furthermore, the aim of this thread is to find a way to encourage all those who rate a level, to rate it fairly and equally across a spectrum of stars. Having one Moderator to cast an opinion of what it should be rated would be an impossibly large task and would end up inevitably leading to disagreements due to personal biases.

    My two cents. Thanks for the input though! :)
     
    philandy likes this.
  19. Abi Irsyad

    Abi Irsyad Active Member

    Messages:
    24
    Levels:
    30
    Albums:
    2
    Likes Received:
    118
    Joined:
    Aug 23, 2018
    I just joined this forum for less than a month, newbie. I saw this thread discussing the rating system. I actually still don't understand about giving stars, 1 - 5, and measuring whether this forum really has that rating?
     
  20. nGord

    nGord Standby Administrator, Retired Moderator Staff Member

    Messages:
    608
    Levels:
    7
    Albums:
    4
    Likes Received:
    2,827
    Joined:
    Jul 21, 2016
    Yes, this forum does have a rating system. It only becomes a problem when members rate only with one number - that's not a rating. A single number is akin to a "like" i.e. it is more of an acknowledgement of having played the level. It's worse behaviour than not rating at all.

    As long as members make at least some effort to differentiate what a 5-star level is compared to a 3-star level for example, then it will have some value. Remember that 3-stars is average and most levels are average by definition so that should be an easy baseline.

    One place that it makes a big difference is in @Gepeto's monthly statistics compilation. If all ratings are always 5-stars, then levels are not ranked by anything more than popularity of the creator. If only one person rates with something other than 5-stars, then it suddenly has a significant value. Thus the use of other than 5-stars must be encouraged by everyone in the community and especially noted to the noobs in level comments.
     

Share This Page