Page 1 of 3

Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 8:58 am
by Lunkhead
There must be a lot of ways to judge songwriting competitions, I bet.

One way that is familiar to many folks around here is the Nur Ein way. The judges each rank the songs in a round individually, most liked song with the highest number ranking, least liked song ranked with a 1. Then the rankings from all the judges are summed together to determine the final ranking.

This has the advantage of being efficient in that the judges do not have to coordinate or communicate at all. They just need to listen to the songs and rank them on their own and submit their rankings. In a situation where that has to be done with 48 hours, that efficiency is important.

Polarized opinions can result in very interesting results, though. Someone can win without any judge ranking them as the winner. And someone can be eliminated without any judge ranking them as being eliminated.

Someone compared this to Olympic judging. You would not be likely to see, for example, a panel of judges there be split the way the Nur Ein judges often are, giving the same gymnastics routine a 10, a 10, and a 1. (Apparently that style of scoring gymnastics got thrown out 10+ years ago but the point isn't about the actual numerical values of the scores but about their relative consistency.)

Sometimes I've thought it would be great if in a tournament like Nur Ein the judges had to collectively pick the winner and the eliminated participants in each round. It would take more time because folks would have to confer to the point of agreement, and it would be more difficult because there'd have to be some compromising and cooperation involved to reach consensus. It's likely not practical to do in the 48-hour time span with judges spread around the country or world.

I'm mostly interested in this right now in the context of an idea I had of trying to figure out a way to bring some of the intensity and participants from Nur Ein over into Song Fight, by coming up with some kind of unofficial "meta" tournament layer on top of Song Fight.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 10:47 am
by Geech
Seems like you're asking two different things - one is how to arrive at more consistency for judging in general and another to bring over more enthusiasm to SF.

I think a conference for judging is unrealistic for the reasons you state, but I also think it could be bad because some judges will be better advocates for their choices simply because they are better at arguing. For something like Nur Ein, the rankings are irrelevant for the purposes of the competition. All you need is a winner and a list of those eliminated. The rankings help arrive at that, true, but for those who do not win and are not eliminated it doesn’t matter. (Now, I actually like the rankings regardless as a listener, as they are generally accurate, with better songs ranked higher.)

I think any scored judging should consider throwing out the top and bottom scores for each submission. This would reduce some of the crazy disparity seen in Nur Ein. I would then keep the rankings. As an artist, it’s good to understand what the judges are looking for. I would require the judges to submit reviews for each submission, either in written or podcast/oral form.

For SF, in comparison to NE, I think that because SF is on-going vs NE being once per year and that it is elimination-oriented over weeks just make it more intense. I also think that having judges makes it seem more serious than whatever SF members and randos over the internet that can vote on in SF.

For SF, maybe there should be two tiers of voting
1. As it is done today, anyone on the internet could vote for any two+.
2. Any SF member could rank them best to worst (either precise rankings or, perhaps more simply, in three tiers), provided she includes written or podcast reviews to explain.

And the voting in the second tier would be worth some amount more than the first. This would have the additional benefit of weighting the voting towards SF members instead of people's random friends and family on the internet.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 11:40 am
by Niveous
Geech wrote:
Thu Jun 06, 2019 10:47 am

For SF, maybe there should be two tiers of voting
1. As it is done today, anyone on the internet could vote for any two+.
2. Any SF member could rank them best to worst (either precise rankings or, perhaps more simply, in three tiers), provided she includes written or podcast reviews to explain.

And the voting in the second tier would be worth some amount more than the first. This would have the additional benefit of weighting the voting towards SF members instead of people's random friends and family on the internet.
Interesting. It's the Rotten Tomatoes way. 1 is the Audience Score, 2 is the Critical Consensus.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 12:43 pm
by ken
I think it would be interesting to create some kind of rubric for judging and see where that gets you. I'm not sure what exactly that would look like, but maybe something like a Creative Writing course rubric you can find online.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 12:45 pm
by Lunkhead
Yeah, I sort of broached two subjects but I only wanted to mention the topic of "[how] to bring over more enthusiasm to SF" to provide context for why I'm curious about "how to arrive at more consistency for judging". I don't want to actually tweak Song Fight itself in any way. And I think discussing what motivates people more to participate in Nur Ein than Song Fight is a whole other topic to try to delve into in a separate thread.

Tossing out the top/bottom rankings is an interesting idea for sure. And we can look at the Nur Ein results and apply that retroactively to see how that would have changed the outcomes of rounds. Maybe one of stats/spreadsheet loving folks would enjoy diving into that analysis. Glenn? Glennny? Niveous? ;)

Audience vs critical evaluation is also interesting and definitely has precedents in not just Rotten Tomatoes but many shows of the reality competition type that if I recall correctly originally inspired Nur Ein. I think unfortunately the Song Fight votes are way too easily gamed to be used in a contest where anybody actually cared about winning, personally. But maybe we could have open voting by a larger group than a judges panel (like folks on the boards) could provide a more trustworthy signal.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 12:47 pm
by Pigfarmer Jr
Ranked voting by board members is a great idea. How it would be weighted with a public vote or implemented code wise are all things I have a great amount of ignorance about.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 12:49 pm
by Lunkhead
ken wrote:
Thu Jun 06, 2019 12:43 pm
I think it would be interesting to create some kind of rubric for judging and see where that gets you. I'm not sure what exactly that would look like, but maybe something like a Creative Writing course rubric you can find online.
Yeah, I was imagining trying to figure something like that out too. Something like:

https://academics.lmu.edu/spee/officeof ... plerubric/

Where you provide folks some individual dimensions along which to score aspects of each song, give them possible scores, with descriptions for each score, and have folks pick the ones that apply the best. The current gymnastics scoring system apparently now is actually the sum of two scores, one for difficulty and one for "execution" (that one starts at 10 and execution issues result in reductions to that score).

https://www.bustle.com/articles/177091- ... han-before

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 12:51 pm
by Lunkhead
"rubric for assessing songwriting" returns some potentially interesting material:

https://www.rcampus.com/rubricshowc.cfm ... ode=P95A79&

https://www.slideshare.net/EmilynMapalo ... ent-rubric

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 2:51 pm
by owl
This is an interesting discussion. On the one hand, I like the idea of a rubric leading to more consistent grading, but trying to come up with what constitutes "good" in this context seems really tricky and arbitrary.

e.g. one of those rubrics has this requirement for song structure: "There must be at least TWO separate verses to it, the chorus must have at least four lines to it." Extra points for three or more verses!
I don't even know how many great songs there have to be out there that don't fit this boring metric. (Plenty of songs do, of course. But plenty don't, too.)

It might make more sense in the context of the Nur Ein challenges, which are definitely already arbitrary and kind of mysterious. But for general judging it just seems like you might end up with a lot of formulaic or box-checking songs floating to the top. Whereas a song that's great but breaks rules or where it's not easy to articulate what you like about it might end up getting thrown out.

I'd like the throwing out the top and bottom scores idea better if there were more judges involved. It seems like as a judge maybe you'd then rank your actual favorites high-but-not-highest in an attempt to protect them? And mediocrity might be rewarded? Or a judge whose tastes don't align with the others' would have their vote voided out more often than not? My husband has a theory that the most polarizing movies on Rotten Tomatoes are really the best ones, he's always way more interested in movies with a bunch of high and low scores than ones with a consistent fairly high score.

The difficulty + execution thing is interesting, maybe scoring on different broad facets of a song would work--not necessarily with strict rules about what constitutes "good" for that category, to give judges some freedom, but individual scores for different aspects to take into consideration plus a "personal ranking" for songs that defy categorization, or are more than the sum of their parts, or particularly speak to a judge for one reason or another?
  • music and structure
  • lyrics and melody [not always applicable?]
  • arrangement and production [how do you rank a G&G song against EDM?]
  • performance
  • etc.
but what is important? Originality? Catchiness? Emotion? Listenability? There are so many good songs that would break any of those metrics too.

On the other hand I guess people may be unconsciously scoring on these types of metrics anyway and this might just make it more transparent.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 3:30 pm
by Lunkhead
e.g. one of those rubrics has this requirement for song structure: "There must be at least TWO separate verses to it, the chorus must have at least four lines to it." Extra points for three or more verses!
I don't even know how many great songs there have to be out there that don't fit this boring metric. (Plenty of songs do, of course. But plenty don't, too.)
Hm, it's possible I misread or too quickly glossed over those two rubrics. Doh! Checking more closely now I guess that first one is more specifically prescriptive than I thought. That's not entirely what I had in mind. I was more thinking about something like the second one where the descriptions for the scores are more qualitative than quantitative.
On the other hand I guess people may be unconsciously scoring on these types of metrics anyway and this might just make it more transparent.
Yeah, that to me seems like a big potential benefit. That and possibly some consistency. I was thinking a rubric could be helpful in two ways, particularly for judges who maybe aren't trained in music or aren't otherwise musically very technical, or maybe are just music lovers and not musicians. They could help to get judges to think about dimensions like performance and arrangement more explicitly than they might have otherwise. And they could help to give some suggested general/non-technical descriptions for scores in categories to help judges match up their feelings about a song with scores. In a sense they'd provide some guide rails and sign posts to attempt to get judges to consider songs in more ways and in more depth.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 3:35 pm
by Lunkhead
I'd be curious about some specific examples folks have too of great songs that break metrics of "Originality Catchiness Emotion Listenability" or otherwise "break the rules". But that's probably a whole other thread's worth of a topic. I have definitely thought of a lot of songs that I love and think are great and been like "That would be eliminated in Round 1 of Nur Ein", lol. I'll have to think of some examples later.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 4:41 pm
by grumpymike
Not to be a downer, but what’s different about this time we all think of ideas that take effort to implement?

Honestly, if the goal is to have somewhat understandable Nur Ein rankings, the easiest thing to do is ban mastering and normalize all of the tracks before handing them to the judges.

If the goal is to have better rankings in general, everyone should be required to judge every X Nur Eins. Perhaps extend the round duration such that judges can put more thought in. How many times have you heard “I would have ranked this one higher if I could go back...”

To make Song Fight more involved, IMO the fights need to have consequences, the titles need to change on time, and the web site needs to join the 90s to foster engagement. On the first note: I think a decaying scoreboard would be great. Something where a good solid 3 entries in a month will put you on the leaderboard.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 4:55 pm
by Lunkhead
grumpymike wrote:
Thu Jun 06, 2019 4:41 pm
the easiest thing to do is ban mastering and normalize all of the tracks before handing them to the judges.
Ha! I was just saying at BSS practice that there everybody should have to use the free YouLean loudness meter plugin and master their tracks to a specification. I don't think that would really changes things much, it would just make things nicer for everyone involved. Judges wouldn't have to reach for the volume knob between songs, participants wouldn't have to read/hear judges complain about that, etc. ;) It might be helpful in terms of maybe setting the specific loudness range a bit lower than the most extreme loudness some folks used, to give everybody's ears a break.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 5:01 pm
by grumpymike
Lunkhead wrote:
Thu Jun 06, 2019 4:55 pm
I don't think that would really changes things much,
From what I’ve seen in judging, yes, certain judges are affected differently and this year I think a lot of songs rode past-or got cut- on mastering. I think that’s part of the beautiful game that makes everything a little more random. But if you want to reduce randomness, easy win.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 5:21 pm
by Chumpy
grumpymike wrote:
Thu Jun 06, 2019 4:41 pm
Honestly, if the goal is to have somewhat understandable Nur Ein rankings, the easiest thing to do is ban mastering and normalize all of the tracks before handing them to the judges.
I think "easiest" may not be the right word here. You might want to consider tackling low hanging fruit like id3 tag normalization before you set your sights on loudness normalization. I do think it would be cool though. You know what else would be cool? Artist accounts and mp3 uploads.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 6:31 pm
by grumpymike
Chumpy wrote:
Thu Jun 06, 2019 5:21 pm
I think "easiest" may not be the right word here. You might want to consider tackling low hanging fruit like id3 tag normalization before you set your sights on loudness normalization. I do think it would be cool though. You know what else would be cool? Artist accounts and mp3 uploads.
I wasn't even talking loudness so much as banning contestants from using a whole mastering FX chain. I agree all of that would be nice to have, though I don't think that solves Sam's perceived "randomness" in the judging. The randomness is a factor of different people noticing different things and weighting them differently - even if unconsciously. So just start eliminating factors! No mastering, no samples, only 2 instruments, only English languages, under two minutes, only an approved subset of words, only 8/8... Hey even the freestyle olympic events have restrictions. I'm being pretty tongue-in-cheek here.

If we want to talk collaborating on code in general, which would be helpful for progress, I think most of the interested parties have discussed it before and turned up their noses at the programming language/priorities/etc. :)

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 7:04 pm
by MicahSommer
grumpymike wrote:
Thu Jun 06, 2019 4:41 pm
If the goal is to have better rankings in general, everyone should be required to judge every X Nur Eins. Perhaps extend the round duration such that judges can put more thought in. How many times have you heard “I would have ranked this one higher if I could go back...”
To me, the short turnaround for judges' rankings is the craziest thing about Nur Ein. SpinTunes gives a week to submit entries and then the judges have a week to rank and review before the next prompt is even posted. Not that Nur Ein has to turn into SpinTunes, but it lets the judges be more thoughtful in their rankings and the participants usually end up with more extensive feedback.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 7:13 pm
by Chumpy
grumpymike wrote:
Thu Jun 06, 2019 6:31 pm
If we want to talk collaborating on code in general, which would be helpful for progress, I think most of the interested parties have discussed it before and turned up their noses at the programming language/priorities/etc. :)
I had envisioned a system where you upload your mp3 to to whatever music contest site, and it runs some LUFS analysis software on it and based on what it discovers either accepts, rejects, or offers to "fix" it. This hypothetical system would know when the deadline for submission is and would let your submit/replace your song as many times as you like prior to the deadline.

I don't know if you've ever used ffmpeg, but it's an amazing Open Source audio and video manipulation library. I use it to automatically generate little mp3 chapter files based on the chapter metadata in our finished podcast mp3 so I can make links to individual chapters. It also has a module called "loudnorm" which can both collect detailed loudness stats on a given audio file, and can also normalize your audio to specific parameters. All the hard code is written, it just needs to be glued together by whatever scripting language you like.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 7:33 pm
by grumpymike
Chumpy wrote:
Thu Jun 06, 2019 7:13 pm
I had envisioned a system where you upload your mp3 to to whatever music contest site, and it runs some LUFS analysis software on it and based on what it discovers either accepts, rejects, or offers to "fix" it.

Yeah, I'm looking at ffmpeg for exactly that and some other related purposes at this very moment. That's the easy part. The annoying part that I'm not interested in tackling - but have frequently thought about - is prioritizing building out a user system.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 7:42 pm
by Lunkhead
Maybe a new thread would be a good place for discussing code and ffmpeg and stuff that's not about how to judge songs ... ? I also don't really think any of that stuff would/should matter that much in judging, I would hope. If any judge were like "Great song, but, this other one that I liked just as much had mp3 art and yours didn't, so, I ranked you lower" they should be immediately fired, imho. :P

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 7:49 pm
by Chumpy
Lunkhead wrote:
Thu Jun 06, 2019 7:42 pm
If any judge were like "Great song, but, this other one that I liked just as much had mp3 art and yours didn't, so, I ranked you lower" they should be immediately fired, imho. :P
Heh, stranger things have happened. It's a pretty well established notion that things that are louder sound better. That's why it's so import to level match plugins when you're A/B testing mix changes. Having contest songs adhere to loudness standards can help level the playing field. I wouldn't be surprised if BYD lost points in some judge's estimation due to their quiet mixes.

Re: Songwriting competition judging mechanisms

Posted: Thu Jun 06, 2019 7:58 pm
by Lunkhead
I wasn't contesting that part, more the stuff about ID3 tags.