Songwriting competition judging mechanisms

Discuss the many little competitions/projects that spring up amongst the Song Fight community.
User avatar
glennny
Jump
Posts: 2196
Joined: Sat Mar 04, 2006 2:39 am
Instruments: Guitar, Bass, Mandolin, Dobro, Banjo, E-Bow, Glock
Recording Method: Garageband
Submitting as: Berkeley Social Scene
Location: Castro Valley, California

Re: Songwriting competition judging mechanisms

Post by glennny »

1. Ban auto-tune
2. get rid of challenges, they ruin creativity and songs
3. Don't let Nur Ein become Spin Tunes, keep Spin Tunes Spin Tunes, keep Nur Ein Nur Ein

but seriously, It'd be nice to know how each judge judges, and what is valuable to them. I'd like to know ahead of time how they hate David Bowie, Alice Cooper and U2. It's nice to know who your audience is when you're writing a song. For most of Nur Ein it's an Audience of 5.
Phillipso, Older Brothers, Semolina Pilchards, Zipline , Thank Glennny for the Frisbee, The Odoriferous Valley, The Worldly Self Assurance, Berkeley Social Scene, Very Gentle Knives, Daddy Bop Swing Set, GUNS, The Kraken Lives, Cavedwellers
mo
Mean Street
Posts: 513
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass, synth
Recording Method: Reaper 4eva
Submitting as: duboce triangle, ellipsis, agony sauce, moody vermin, spite, yaks
Pronouns: n/a
Location: hell a
Contact:

Re: Songwriting competition judging mechanisms

Post by mo »

Hey Sam, as someone who used to design game structures for a living, I wanted to ask two questions before I contribute my thoughts:

1. How do you define "consistency" in the judging?
2. What do you find desirable about increased consistency, as you define it, in the judging?

Thanks!
User avatar
Spintown
Push Comes to Shove
Posts: 399
Joined: Fri Oct 02, 2009 9:27 pm
Submitting as: Spintown & Company
Pronouns: he/him
Contact:

Re: Songwriting competition judging mechanisms

Post by Spintown »

I like the idea of allowing forum members (maybe long time members given a rank?) to vote on Nur Ein as long as the votes aren't public until the results are in. When doing SpinTunes the hardest thing for me was getting judges & guest judges willing to do reviews in the time they were given.
I always felt pressure to make sure I had at least 5 different people posting feedback. If I had it set up so that the community could rank the songs I'd have been a little less stressed because even if they didn't post reviews, more people ranking your songs IS feedback. And in general, the more opinions you get the more likely the better songs float to the top.
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

mo wrote:
Sun Jun 09, 2019 5:12 pm
Hey Sam, as someone who used to design game structures for a living, I wanted to ask two questions before I contribute my thoughts:

1. How do you define "consistency" in the judging?
2. What do you find desirable about increased consistency, as you define it, in the judging?

Thanks!
Good questions! Thanks for asking.

1. I mean less wild deviation between the judges rankings/scores for each song, primarily at the top and bottom. Ideally fewer or preferably no times where anyone got ranked e.g. top by three judges and bottom by two, etc.
2. What I find desirable about that is the hope that the results might seem less like the roll of the dice/the average of somewhat random numbers, and more like the result of a considered, collective evaluation of the entries.
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: Songwriting competition judging mechanisms

Post by jb »

In the case of a Nur Ein situation, though this topic wasn’t about that originally, would the goal be to receive not only a more objective critique from the judges, but for that objectivity to carry weight in the outcome?

Tough to achieve since so many measures of songcraft are inherently subjective— creativity, performance effectiveness, originality. Beauty, humor, pathos, romance, tragedy. And the nature of both NE and SF are anything goes as far as genre and topic.
blippity blop ya don’t stop heyyyyyyyyy
mo
Mean Street
Posts: 513
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass, synth
Recording Method: Reaper 4eva
Submitting as: duboce triangle, ellipsis, agony sauce, moody vermin, spite, yaks
Pronouns: n/a
Location: hell a
Contact:

Re: Songwriting competition judging mechanisms

Post by mo »

See I basically disagree with the premise that the wild swings indicate something undesireable. I think that it proves that the judges aren’t being swayed by groupthink and that each judge has individual preferences/tastes/criteria.

In the case of something Frankie’s exit, where he got two 2nd place votes and 2 second to last place votes and got eliminated, that’s a situation that I think could stand to be reviewed. On the one hand, it’s a song that polarized the judges and therefore it is fair that it not get a top spot, but probably it should have passed, everything else being equal.

I’ve thought about it a little, and I’m sure this discussion has been had before, but I do think the judges need to all deliver a statement of their judging philosophy, musical tastes, and how they intend to approach the challenges, before the competition so that people have time to understand and digest them. If there was such a thing this time, I missed it, which could entirely be my fault, but if there wasn’t, it should be more prominent. That would help a lot, because as glennny noted, you make the songs fro the judges, not for only an audience of people who like your niche. I would hope that clearer information would make a huge difference.

The other information that would make a huge difference is knowing how the challenges are going to be evaluated. I’ve seen a little confusion sometimes as to what the letter vs the spirit of the law is, as it were, and so more specificity as to what the range is might help.

Finally in terms of actual systems, the easiest thing to do in order to prevent outcomes that feel “unjust” is simply to weight the top. In Nur Ein, after round Zero you could give a top rank a +3 point value, 2nd rank +2, 3rd rank +1, everything else as same. That has the effect of making getting a top rank more valuable because just having one top rank could give you enough extra points to stave off elimination if someone else ranked you bottom. In the situation where you got all 1s and one top rank, it probably wouldn’t save you (depending on the round, I haven’t done all the math—I’m not suggesting this exact weighting, but just giving an example). It would help in making sure a song that got 3rd and 4th ranks but no 1st ranks would have a harder time winning a round against a song that got three top ranks for example.

Just a few thoughts, hope they give someone else some ideas.
User avatar
ujnhunter
Ice Cream Man
Posts: 1804
Joined: Fri Mar 07, 2008 1:09 pm
Instruments: Bass, Keyboards, Crummy Guitar & Animal Noises (especially Donkeys)
Recording Method: Reaper 5.9x, Tascam FireOne/Behringer UMC202HD/Avid Eleven Rack/Line 6 UX2, Win 7 PC / Win 10 Laptop
Submitting as: Cock, Chth*.*, D.A.H. (Der Alter Hahn)
Pronouns: His Infernal Majesty
Location: CT, USA
Contact:

Re: Songwriting competition judging mechanisms

Post by ujnhunter »

Who's to say that the judges that ranked you bottom aren't correct? Why should someone be saved by default and not eliminated by default?
-Ujn Hunter
Photovoltaik - Free 6 Track EP - Song Fight! Liner Notes
Billy's Little Trip wrote:I must have this....in my mouth.....now.
User avatar
glennny
Jump
Posts: 2196
Joined: Sat Mar 04, 2006 2:39 am
Instruments: Guitar, Bass, Mandolin, Dobro, Banjo, E-Bow, Glock
Recording Method: Garageband
Submitting as: Berkeley Social Scene
Location: Castro Valley, California

Re: Songwriting competition judging mechanisms

Post by glennny »

Finally in terms of actual systems, the easiest thing to do in order to prevent outcomes that feel “unjust” is simply to weight the top. In Nur Ein, after round Zero you could give a top rank a +3 point value, 2nd rank +2, 3rd rank +1, everything else as same......
I'd like to bet on Rain Watt to win
Max Bombast to place
and Rachel to show

the closer the songs are in quality the higher potential for wild swings in rankings. They're all good signals, the strength is in the noise.

I grew to really dislike challenges that took me away from writing a song. In my opinion there was way too much time spent on dealing with other peoples songs, rather than trying to write an original song.
Phillipso, Older Brothers, Semolina Pilchards, Zipline , Thank Glennny for the Frisbee, The Odoriferous Valley, The Worldly Self Assurance, Berkeley Social Scene, Very Gentle Knives, Daddy Bop Swing Set, GUNS, The Kraken Lives, Cavedwellers
User avatar
Pigfarmer Jr
Jump
Posts: 2293
Joined: Sat Mar 21, 2009 6:13 am
Instruments: Guitar
Recording Method: Br-900CD and Reaper to mix
Submitting as: Pigfarmer Jr, Evil Grin, Pork Producer, Gilmore Lynette Tootle, T.C. Elliott
Pronouns: he/him
Location: Columbia, Missouri
Contact:

Re: Songwriting competition judging mechanisms

Post by Pigfarmer Jr »

I think there is a difference in a song with one low ranking and the rest very high vs a song with scattered rankings or multiple high and low rankings. And honestly, the most intriguing songs are the ones that have high and low scores. I wonder how many songs I truly love that friends have hated (and vice versa.) Especially on the first listen or three.

In the example above, kicking out the high and low score would "help" the song with one low score and the rest mid to high scores. It wouldn't do anything for the song with two first place votes a middle vote and two last place votes. Which makes a lot of sense to me.

I really like the idea of SF songs being ranked as a score. But it takes longer to vote that way. Is that a good or bad thing? I'm not sure it's either. But those folks who review the songs wouldn't have *that* much more work to do to rank them. Those of us who don't always review would have to spend more time to rank them. But the added bonus is that maybe we'll get more reviews (or I'll be more consistent in posting reviews?) And it doesn't address the discrepancy in rankings that we're talking about.
Evil Grin bandcamp - Evil Grin spotify
T.C. Elliott bandcamp - T.C. Elliott spotify

"PigFramer: Guy and guitar OF MY NIGHTMARES." - Blue Lang
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

glennny wrote:
Tue Jun 11, 2019 6:22 am
Finally in terms of actual systems, the easiest thing to do in order to prevent outcomes that feel “unjust” is simply to weight the top. In Nur Ein, after round Zero you could give a top rank a +3 point value, 2nd rank +2, 3rd rank +1, everything else as same......
I'd like to bet on Rain Watt to win
Max Bombast to place
and Rachel to show
Maybe you were joking but https://en.wikipedia.org/wiki/Ranked_voting might be an interesting option.

(Also just to be clear my intended topic here is not "What's wrong with Nur Ein" "How to change Nur Ein" etc. Just brainstorming for more different ideas because I'm dreaming of trying to operate something similar to Nur Ein, on top of Song Fight, unofficially (just as myself not as a fightmaster), but with some differences in the judging mechanics.(
mo
Mean Street
Posts: 513
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass, synth
Recording Method: Reaper 4eva
Submitting as: duboce triangle, ellipsis, agony sauce, moody vermin, spite, yaks
Pronouns: n/a
Location: hell a
Contact:

Re: Songwriting competition judging mechanisms

Post by mo »

Yeah I mean we have ranked voting now in Nur Ein, more or less (ie not popular but with judges), and as a voting system it functions fine, IMO. No system that involves humans is going to be objective or foolproof, I think we all agree, so given that, the system itself is mostly ok.

If you want a different system for something else, I would suggest doing some test runs of the Olympic model you guys were discussing earlier. So something like 1-10 points assigned for categories like songcraft, originality, arrangement, mixing/mastering, vocal performance, personal. Where personal can be whatever the judge personally feels. It would be a max of 60 points per voter, so if you culls, you’d need to have good tiebreakers, OR a system such that everyone who gets a certain number of points passes. If everyone is over the line, everyone passes. Then you just keep moving the line up as rounds go on. So you could make it cumulative, so your score in the previous rounds matters beyond just advancement.then the highest points at the end of however many rounds wins. Just throwing it out there.
User avatar
Pigfarmer Jr
Jump
Posts: 2293
Joined: Sat Mar 21, 2009 6:13 am
Instruments: Guitar
Recording Method: Br-900CD and Reaper to mix
Submitting as: Pigfarmer Jr, Evil Grin, Pork Producer, Gilmore Lynette Tootle, T.C. Elliott
Pronouns: he/him
Location: Columbia, Missouri
Contact:

Re: Songwriting competition judging mechanisms

Post by Pigfarmer Jr »

For a nur ein type of competition, the idea of cumulative voting is appealing, but maybe more in the idea than in practice (possibly.)
Evil Grin bandcamp - Evil Grin spotify
T.C. Elliott bandcamp - T.C. Elliott spotify

"PigFramer: Guy and guitar OF MY NIGHTMARES." - Blue Lang
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

mo wrote:
Tue Jun 11, 2019 11:00 am
Yeah I mean we have ranked voting now in Nur Ein, more or less (ie not popular but with judges), and as a voting system it functions fine, IMO. No system that involves humans is going to be objective or foolproof, I think we all agree, so given that, the system itself is mostly ok.
The judges rank every entry but then the rankings are just summed. We don't have a ranked choice voting system where an entry that gets a majority of first place votes would win, or if no entry got a majority of first place votes, then second-place rankings would be considered, etc.

https://ballotpedia.org/Ranked-choice_voting_(RCV)
https://en.wikipedia.org/wiki/Instant-runoff_voting

That could be complicated to tally up but there are tools out there: https://petertheone.github.io/IRV/ and it could maybe be applied "in reverse" to determine who gets eliminated?
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead »

OR a system such that everyone who gets a certain number of points passes. If everyone is over the line, everyone passes. Then you just keep moving the line up as rounds go on. So you could make it cumulative, so your score in the previous rounds matters beyond just advancement.then the highest points at the end of however many rounds wins. Just throwing it out there.
Those are definitely interesting ideas! I think SpinTunes maybe experimented with that idea, of folks continuing to participate rather than being eliminated, with the final results being cumulative across all the rounds. Spin or glennny, did that happen?
mo
Mean Street
Posts: 513
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass, synth
Recording Method: Reaper 4eva
Submitting as: duboce triangle, ellipsis, agony sauce, moody vermin, spite, yaks
Pronouns: n/a
Location: hell a
Contact:

Re: Songwriting competition judging mechanisms

Post by mo »

Oh, ranked choice voting. Yeah, that tends to work better when you have lots of votes, so if you're planning to do something with a popular vote instead of a limited judging pool, that could be a great way to go.
mo
Mean Street
Posts: 513
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass, synth
Recording Method: Reaper 4eva
Submitting as: duboce triangle, ellipsis, agony sauce, moody vermin, spite, yaks
Pronouns: n/a
Location: hell a
Contact:

Re: Songwriting competition judging mechanisms

Post by mo »

Another suggestion with the points categories is that of course categories can be worth different amounts of points, so let's say songcraft and originality are worth 10 points maximum each, but maybe mixing/mastering is worth 5 maximum, etc. My big concern with this system is that in any one round, you're likely to have a whole bunch of ties, or the possibility that every song is pretty good, which is why I suggested just having a pass/fail line. The reason for increasing the threshold as rounds go on is to make it more difficult, so that only songs of a certain quality will get through--but of course that is still the totality of the submitted song. Maybe for the final heats such as semi-final and final, you could then also add in ranked choice voting from judges so that there's a clear mechanism for winning/losing. Maybe?
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Songwriting competition judging mechanisms

Post by fluffy »

Okay so back to the question about voting systems: I'm a fan of marking things as "good" or "bad" and letting the overall ratio determine a winner. Mathematically it's similar to what we have on Song Fight! right now but the psychology changes the equation a bit. Maybe each vote could also have a +1 bonus for "favorite song" or something. CoverFight! back in the day had a voting system sorta like that except the "favorite" and "good/bad" were separated which caused some really weird scenarios (like that time that my entry had the highest rating but 0 votes because everyone liked it but it was nobody's favorite).

Ranked-choice/instant-runoff voting is great if you want to spend a lot of time deciding your exact order of preference but that can be difficult in a context where you have a lot of different things which are good or bad on different merits.

Once upon a time I was thinking of ways of doing mini-bracket-style voting, where people are given random pairings of things and asked which one is better between the two, and then trying to figure out the overall sort order based on which ones win in which pairings most often, although that ends up causing all sorts of non-transitive properties that make an objective answer difficult to come by (the classic Rock-Paper-Scissors situation).

Back in the day, GarageBand (the website, not the Apple product) used random-pairings comparisons to handle their challenge voting, although I have no idea how useful it was in the end or what the underlying final ordering was. There's probably a reason nobody remembers GarageBand.
User avatar
ken
Hot for Teacher
Posts: 3869
Joined: Sat Sep 25, 2004 6:10 pm
Instruments: Guitar, bass, drums, keys
Recording Method: MOTU 828x, Cubase 10
Submitting as: Ken's Super Duper Band 'n Stuff
Pronouns: he/him
Location: oakland, ca
Contact:

Re: Songwriting competition judging mechanisms

Post by ken »

fluffy wrote:
Tue Jun 11, 2019 5:09 pm
nobody remembers GarageBand.
Garageband was great because if you did a certain number of reviews you got a gift card to buy a CD of your choosing from some Canadian record store I can remember. That was totally worthwhile.
Ken's Super Duper Band 'n Stuff - Berkeley Social Scene - Tiny Robots - Seamus Collective - Semolina Pilchards - Cutie Pies - Explino! - Bravo Bros. - 2 from 14 - and more!

i would just like to remind everyone that Ken eats kittens - blue lang
mo
Mean Street
Posts: 513
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass, synth
Recording Method: Reaper 4eva
Submitting as: duboce triangle, ellipsis, agony sauce, moody vermin, spite, yaks
Pronouns: n/a
Location: hell a
Contact:

Re: Songwriting competition judging mechanisms

Post by mo »

ken wrote:
Wed Jun 12, 2019 9:04 am
fluffy wrote:
Tue Jun 11, 2019 5:09 pm
nobody remembers GarageBand.
Garageband was great because if you did a certain number of reviews you got a gift card to buy a CD of your choosing from some Canadian record store I can remember. That was totally worthwhile.
What's a CD?
User avatar
BoffoYux
Panama
Posts: 908
Joined: Tue Jan 11, 2011 9:22 pm
Instruments: Keys, Clunking, SFX and Strings
Recording Method: Audacity, Adobe, and other 'A' titled software
Submitting as: Boffo Yux Dudes
Location: New England
Contact:

Re: Songwriting competition judging mechanisms

Post by BoffoYux »

Chumpy wrote:
Thu Jun 06, 2019 7:49 pm
I wouldn't be surprised if BYD lost points in some judge's estimation due to their quiet mixes.
THAT'S the reason why we lost points! I thought it was our acid wit and rogueish good looks...

(However, we will start to tweak the mixes further next time. Thanks for the thoughts on what we've been ignoring all these years.)
User avatar
Æpplês&vØdkã
Push Comes to Shove
Posts: 410
Joined: Thu Jul 27, 2006 1:59 pm
Instruments: Drums, bass, synths, guitar, whatever else is around.
Recording Method: Tascam dp006, Shure SM-58, FL studio
Submitting as: Phlebia
Location: Near Atlanta
Contact:

Re: Songwriting competition judging mechanisms

Post by Æpplês&vØdkã »

Here's a question: and this is mostly in response to the comment about "enthusiasm being dampened by fights going a few days past the deadline before they're posted":

How hard would it be to implement a system where people can link their account to an arbitrary band name (with multiples allowed of course), upload as many revisions to their entry before the deadline, and then as soon as the deadline hits everything is automatically posted from the submission pool?

I don't say that as a critique of the fightmasters, who do a wonderful job of coming up with clever themes and keeping the site running. I say that from the perspective of someone who spends most of my time at work automating systems so people have less to worry about.

I do recognize "account linked submissions" would change the way that this place has worked for 19 years now, but I do think the benefits of having automated posting, an upload mechanism that runs through the site itself rather than external email, and giving users the ability to re-upload their own tracks without pestering fightmaster would be pretty useful.

I have no idea how this site is implemented and how hard that would be. Anyway, just some ideas...
I'm afraid this one fails on pretty much every level for me. - Jim of Seattle

My bandcamp! https://phlebia.bandcamp.com
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Songwriting competition judging mechanisms

Post by fluffy »

We (meaning the admins — JB, Lunkhead, and myself) have definitely talked a bunch about how to make the submission process a bit more automated and data-driven but there’s a bunch of things that make it difficult. Like the current state of the code is incredibly piecemeal and it would take a complete overhaul to get it to a point where it’s be easy to experiment with more features.

There’s also historically been a feeling that if it seems like someone is on the other end of the submission process, people are generally going to be less shitty. If the submission process is automated it’s a lot more likely for people to try to break it. We already have enough trouble with people submitting random songs that don’t belong to them or which weren’t written for the fight, or which are outright trolling.

There’s also some stuff which can’t really be automated away, like writing the flavor text for the winner of the previous fight. So there will always be something that will be a gating factor if the nightmares doesn’t have time to push the update button.

That said, a lot of the process can certainly be improved and made easier, and it would be great if the site had some sort of actual database driving the relations between songs and artists. But it’s also easy to go overboard in trying to design the ultimate system which would solve problems that don’t actually exist.

The current codebase and process have evolved over the past... when did Spud start MeanWhile? 2002? So, around 17 years. There’s a lot of technical debt, putting it mildly.
Post Reply