How should entries be ranked?
category: parties [glöplog]
it's called voting, not prediction :)
Clearly, we need to train a neural network on the video capture to understand your normal voting style, and then you don't have to vote at all and can focus on the all-important basket compo instead.
Quote:
Ok. Question is, which party is going to want to test it?
I agree with Kusma: the first step should be more clarity/transparency in how the current system work.
I find it very problematic that so much sceners (including myself) don't really realize how voting (at a partymeister demoparty) works ... while party results are kind of a big deal (especially for Revision/Evoke).
It should be something superobvious for everyone ; for example everybody knows prods shown at a demoparty were not released before.
Maybe you could try to make "voting education" a central thing of the next Revision?
Don't change anything in the system logic but you make it crystal clear that:
- prods are ranked by the SUM of vote.
- 1 star means "+1", 2 star means "+2"...
Imho you should even drop the stars UI, because stars suggest "rate" and not "popularity vote".
The UI should convey that when you vote you are "giving points to the prod".
- voting is important ; if you don't vote for a prod you liked, you are penalising it
(which may discourage the producer of this prod in the long terme => less of the prods you like)
Maybe it won't change anything, but I know I will personally feel more comfortable with the end-results if there is proper education :p.
I dislike any type of prediction (including bayesian average tbh) because it takes control away from the voter - at least for Revision.
It's easy to change the stars to dots (or points) to make it more clear and to add some explanatory text on the page an in the app to clarify. It might help in distinguishing from Amazon-like star reviews which usually calculate the average. I agree with that analysis.
KeyJ: I wouldn't say that "anything is better than the current method" given that the system with adding values exists since the beginning of demoparties with a few controversies (namevoting, some rank-flipping because of party atmosphere vs. quality of the entries) over 20+ years.
For me it sounds that some people are not happy with the result of one compo. It's good that it sparked this conversation, but I don't know if this is enough to change a system that worked long before Partymeister was born :-)
In the end it's up to the organizers what they want to do. I'll see to it that, same as with Wuhu, that organizers can extend / adapt the system they want to use.
It's easy to change the stars to dots (or points) to make it more clear and to add some explanatory text on the page an in the app to clarify. It might help in distinguishing from Amazon-like star reviews which usually calculate the average. I agree with that analysis.
KeyJ: I wouldn't say that "anything is better than the current method" given that the system with adding values exists since the beginning of demoparties with a few controversies (namevoting, some rank-flipping because of party atmosphere vs. quality of the entries) over 20+ years.
For me it sounds that some people are not happy with the result of one compo. It's good that it sparked this conversation, but I don't know if this is enough to change a system that worked long before Partymeister was born :-)
In the end it's up to the organizers what they want to do. I'll see to it that, same as with Wuhu, that organizers can extend / adapt the system they want to use.
Since there seems to be some confusion about what I meant with "vote prediction", I'll try to explain again.
If you vote for every entry in a compo, these votes are of course counted as you have cast them.
If you don't vote for any entries in a compo, no votes are counted from you in that compo at all.
The interesting part is if you vote for some entries in a compo, but not for all. The votes for the entries you did vote for will of course be counted verbatim. As for the other (unvoted) entries, many users would like to have the voting system interpret this as a mandate to trust the other voters' judgment. This can be done by simply ignoring the (non-)vote when computing an average score, or by predicting or interpolating (if you prefer this term) from other user's votes.
The benefit of the interpolation/prediction method is that it makes life harder for vote fraudsters who'd like to push the ranking of very few specific entries.
So far, with sum-based ranking, it was very easy for them, as they only needed to vote for the entries they wanted to win, thereby automatically penalizing all others.
With average-based ranking, there would no longer be an extra penalty for the unvoted entries, but the fraudsters' target entries' votes would still be pushed too high in the ranking due to the many 5-star votes, compared to the other entries that didn't get those.
With interpolation/prediction, the unvoted entries would also get a rather high score, leveling the playing field quite a bit. (It's enough to dethrone some Evoke 2019 entries that benefited from voting fraud!) The fraudsters would be forced to vote for all entries (5 stars for their targets, 1 star for all others) to circumvent this, which is (depending on the compo) 3 to 20 times as much work as they have to do now. They may still want to manipulate votes, but then they'd have to fucking earn it.
If you vote for every entry in a compo, these votes are of course counted as you have cast them.
If you don't vote for any entries in a compo, no votes are counted from you in that compo at all.
The interesting part is if you vote for some entries in a compo, but not for all. The votes for the entries you did vote for will of course be counted verbatim. As for the other (unvoted) entries, many users would like to have the voting system interpret this as a mandate to trust the other voters' judgment. This can be done by simply ignoring the (non-)vote when computing an average score, or by predicting or interpolating (if you prefer this term) from other user's votes.
The benefit of the interpolation/prediction method is that it makes life harder for vote fraudsters who'd like to push the ranking of very few specific entries.
So far, with sum-based ranking, it was very easy for them, as they only needed to vote for the entries they wanted to win, thereby automatically penalizing all others.
With average-based ranking, there would no longer be an extra penalty for the unvoted entries, but the fraudsters' target entries' votes would still be pushed too high in the ranking due to the many 5-star votes, compared to the other entries that didn't get those.
With interpolation/prediction, the unvoted entries would also get a rather high score, leveling the playing field quite a bit. (It's enough to dethrone some Evoke 2019 entries that benefited from voting fraud!) The fraudsters would be forced to vote for all entries (5 stars for their targets, 1 star for all others) to circumvent this, which is (depending on the compo) 3 to 20 times as much work as they have to do now. They may still want to manipulate votes, but then they'd have to fucking earn it.
Quote:
I dislike any type of prediction [...] because it takes control away from the voter
The very point of this is that it only "takes control away from the voter" where they specifically give away control -- by not voting for entries they didn't see or remember.
Quote:
the system with adding values exists since the beginning of demoparties with a few controversies (namevoting, some rank-flipping because of party atmosphere vs. quality of the entries) over 20+ years.
That doesn't automatically make it a good system :) As we have seen, lots of people misunderstood it badly, and expected to have a neutral voting option all along the way.
I can see the appeal of the "sum voting is a long-standing demoscene tradition, no need to change that" argument though.
[quote]
The very point of this is that it only "takes control away from the voter" where they specifically give away control -- by not voting for entries they didn't see or remember.
If they know that they do that. Maybe they really don't want the entry to get their points. I do that sometimes in compos where I don't like things because I don't think they deserve even one point.
Which brings us back to voter education. Assuming they give away their control might not be ok either.
Quote:
I dislike any type of prediction [...] because it takes control away from the voter
The very point of this is that it only "takes control away from the voter" where they specifically give away control -- by not voting for entries they didn't see or remember.
If they know that they do that. Maybe they really don't want the entry to get their points. I do that sometimes in compos where I don't like things because I don't think they deserve even one point.
Which brings us back to voter education. Assuming they give away their control might not be ok either.
what about dropping the worst and best votes like in figure skating? They drop about 2/8 of the votes, to avoid extremities (and the frauds mentioned by KeyJ).
ps: they also give separated technical and artistic points in some sports. this could also be interesting for demoscene prods :)
ps: they also give separated technical and artistic points in some sports. this could also be interesting for demoscene prods :)
Quote:
ps: they also give separated technical and artistic points in some sports. this could also be interesting for demoscene prods :)
Demodays did that. Wasn't that successful from what I remember.
Quote:
The benefit of the interpolation/prediction method is that it makes life harder for vote fraudsters who'd like to push the ranking of very few specific entries.
So far, with sum-based ranking, it was very easy for them, as they only needed to vote for the entries they wanted to win, thereby automatically penalizing all others.
With average-based ranking, there would no longer be an extra penalty for the unvoted entries, but the fraudsters' target entries' votes would still be pushed too high in the ranking due to the many 5-star votes, compared to the other entries that didn't get those.
With interpolation/prediction, the unvoted entries would also get a rather high score, leveling the playing field quite a bit. (It's enough to dethrone some Evoke 2019 entries that benefited from voting fraud!) The fraudsters would be forced to vote for all entries (5 stars for their targets, 1 star for all others) to circumvent this, which is (depending on the compo) 3 to 20 times as much work as they have to do now. They may still want to manipulate votes, but then they'd have to fucking earn it.
This is a flawed argument from the jump based on this idea that bizarro results are because of fraud.
If you wanna continue this argument, I'd like to hear about instances of this fraud, and names of people who were involved.
By the way, here's a fun scenario to throw a wrench into average-based voting: Compo starts without announcement, first entry is only seen by two people who like it (maybe they're new and don't know what it is) and vote 5, everyone else arrives back in the hall for entries 2-3 and don't vote on the first one. First entry ends up having only 2 votes but 5.0 average.
Quote:
If they know that they do that. Maybe they really don't want the entry to get their points.
Well, currently, the "give 0 out of 5 points" option is labelled "not voted" or simply "-". I'd bet that most people would read this as "not voted = neutral, no harm done".
But OK, we all agree that this is misleading and needs to change. Going away from Amazon-like stars and using an explicit "0 / +1 / ... / +5" scale instead is a good move.
However, I'm still strongly in favor of having an option to vote neutrally, or "give away control", if you want to put it that way. I still think that this should be the default, but it would be OK too if it isn't, as long as such an option exists at all. I want a possibility to vote "had to pee, couldn't watch".
Quote:
this idea that bizarro results are because of fraud
I can say that the most bizarre result of Evoke 2019 ("Eisenerz" on 5th place) is not related to fraud. Instead, the most likely cause is a fluke in live voting, which any alternate vote evaluation model that I tried would have corrected.
However, there are several entries that profited from the extra ~150(!) vote points they got, and ranked higher than they may have deserved.
Quote:
Compo starts without announcement, first entry is only seen by two people who like it (maybe they're new and don't know what it is) and vote 5, everyone else arrives back in the hall for entries 2-3 and don't vote on the first one. First entry ends up having only 2 votes but 5.0 average.
Yes, and the theoretical threat of such a scenario is exactly why I started a call to party organizers to give me (anonymized) voting data earlier in this thread, so we can see if such a thing may actually happen, or if we're generally off by an order of magnitude. (For example, no entry at Evoke 2019 got less than 24 votes; at Evoke 2017, the lower bound was 49 votes.)
Quote:
By the way, here's a fun scenario to throw a wrench into average-based voting: Compo starts without announcement, first entry is only seen by two people who like it (maybe they're new and don't know what it is) and vote 5, everyone else arrives back in the hall for entries 2-3 and don't vote on the first one. First entry ends up having only 2 votes but 5.0 average.
Throws a wrench into average-across-voters-based with no confidence intervals indeed, does not affect any of the other suggestions in the thread. In particular:
- In variations with confidence intervals, they will be very wide, so entry will rank low
- In variations where unvoted entries get the average of that voter's score, will rank (potentially just above) middle
"Ranked pairs" and similar methods should also reduce the susceptibility to fraud since these methods only consider preferences for each pair of voted entries, not absolute votes.
Example: entry X got 4 stars and entry Y got 5 stars in all legitimate votes; with sum-based ranking anyone using 30 keys to boost entry X will outweigh about 150 legitimate votes whereas with ranked pairs they could only outweigh 30 legitimate votes.
Example: entry X got 4 stars and entry Y got 5 stars in all legitimate votes; with sum-based ranking anyone using 30 keys to boost entry X will outweigh about 150 legitimate votes whereas with ranked pairs they could only outweigh 30 legitimate votes.
for those who run partymeister and want to see averages in results, use this diff:
Code:
diff -r 1260323fc8b3 modules/project/views/backend/voting/ascii.tpl
--- a/modules/project/views/backend/voting/ascii.tpl Fri Aug 29 23:18:57 2014 +0400
+++ b/modules/project/views/backend/voting/ascii.tpl Mon Sep 02 11:38:57 2019 +0300
@@ -1,4 +1,4 @@
- .______. ._____. .___ ____. .__________________________|\
+ .______. ._____. .___ ____. .__________________________|\
._\|_ |__\|_ |_\|_ /___._/ |/___\|_ \
| / | / | _/ | _/ / partymeister v2.00 \
|_ /______| _ | \ | | -------------------- - /
@@ -14,8 +14,8 @@
{foreach item=competition from=$competitions}
{$competition->name|indent:2}
- -------------------------------------------------------------------------- -
-{foreach item=entry key=key from=$competition->entries->find_all_by_rank()}{if $entry->is_remote}{assign var='name' value="{utf8_decode($entry->title)} by {utf8_decode($entry->author)} [REMOTE]"}{else}{assign var='name' value="{utf8_decode($entry->title)} by {utf8_decode($entry->author)}"}{/if}
-{($entry->rank)|string_format:'%02d'|indent:7} {$entry->points|number_format:0|string_format:'%4s'} {$name|wordwrap:65:"\n "}
+{foreach item=entry key=key from=$competition->entries->find_all_by_rank()}{assign var='average' value="{($entry->points)/($entry->voted)}"}{if $entry->is_remote}{assign var='name' value="{($entry->title)} by {($entry->author)} [REMOTE]"}{else}{assign var='name' value="{($entry->title)} by {($entry->author)}"}{/if}
+{($entry->rank)|string_format:'%02d'|indent:7} {$entry->points|number_format:0|string_format:'%4s'} {$name|wordwrap:65:"\n "} ( shown {$entry->sort_position|number_format:0}, {$entry->voted|number_format:0} votes, avg={$average|number_format:3} )
{/foreach}
{/foreach}
yeah, we're not falling for the "friendly Russian guy has some code to add to your voting system" trick again
Especially on a party system that's not public.
¯\_(ツ)_/¯