Songwriting competition judging mechanisms

Discuss the many little competitions/projects that spring up amongst the Song Fight community.
User avatar
glennny
Loris
Posts: 1978
Joined: Sat Mar 04, 2006 2:39 am
Instruments: Guitar, Bass, Mandolin, Dobro, Banjo, E-Bow, Glock
Recording Method: Garageband
Submitting as: Berkeley Social Scene
Location: Castro Valley, California

Re: Songwriting competition judging mechanisms

Post by glennny » Thu Jun 06, 2019 11:05 pm

1. Ban auto-tune
2. get rid of challenges, they ruin creativity and songs
3. Don't let Nur Ein become Spin Tunes, keep Spin Tunes Spin Tunes, keep Nur Ein Nur Ein

but seriously, It'd be nice to know how each judge judges, and what is valuable to them. I'd like to know ahead of time how they hate David Bowie, Alice Cooper and U2. It's nice to know who your audience is when you're writing a song. For most of Nur Ein it's an Audience of 5.
Phillipso, Older Brothers, Semolina Pilchards, Zipline , Thank Glennny for the Frisbee, The Odoriferous Valley, The Worldly Self Assurance, Berkeley Social Scene, Very Gentle Knives, Daddy Bop Swing Set, GUNS, The Kraken Lives, Cavedwellers
mo
Marmoset
Posts: 189
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass
Recording Method: behringer umc202hd, guitar rig, cubase, garageband, audition, reason
Submitting as: duboce triangle
Location: hell a

Re: Songwriting competition judging mechanisms

Post by mo » Sun Jun 09, 2019 5:12 pm

Hey Sam, as someone who used to design game structures for a living, I wanted to ask two questions before I contribute my thoughts:

1. How do you define "consistency" in the judging?
2. What do you find desirable about increased consistency, as you define it, in the judging?

Thanks!
User avatar
Spintown
Tamarin
Posts: 392
Joined: Fri Oct 02, 2009 9:27 pm
Submitting as: Spintown
Contact:

Re: Songwriting competition judging mechanisms

Post by Spintown » Sun Jun 09, 2019 7:12 pm

I like the idea of allowing forum members (maybe long time members given a rank?) to vote on Nur Ein as long as the votes aren't public until the results are in. When doing SpinTunes the hardest thing for me was getting judges & guest judges willing to do reviews in the time they were given.
I always felt pressure to make sure I had at least 5 different people posting feedback. If I had it set up so that the community could rank the songs I'd have been a little less stressed because even if they didn't post reviews, more people ranking your songs IS feedback. And in general, the more opinions you get the more likely the better songs float to the top.
Lunkhead
Gorilla
Posts: 6069
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/reason/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead » Mon Jun 10, 2019 8:06 pm

mo wrote:
Sun Jun 09, 2019 5:12 pm
Hey Sam, as someone who used to design game structures for a living, I wanted to ask two questions before I contribute my thoughts:

1. How do you define "consistency" in the judging?
2. What do you find desirable about increased consistency, as you define it, in the judging?

Thanks!
Good questions! Thanks for asking.

1. I mean less wild deviation between the judges rankings/scores for each song, primarily at the top and bottom. Ideally fewer or preferably no times where anyone got ranked e.g. top by three judges and bottom by two, etc.
2. What I find desirable about that is the hope that the results might seem less like the roll of the dice/the average of somewhat random numbers, and more like the result of a considered, collective evaluation of the entries.
User avatar
jb
Lemur
Posts: 3672
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Location: WASHINGTON, DC
Contact:

Re: Songwriting competition judging mechanisms

Post by jb » Mon Jun 10, 2019 9:02 pm

In the case of a Nur Ein situation, though this topic wasn’t about that originally, would the goal be to receive not only a more objective critique from the judges, but for that objectivity to carry weight in the outcome?

Tough to achieve since so many measures of songcraft are inherently subjective— creativity, performance effectiveness, originality. Beauty, humor, pathos, romance, tragedy. And the nature of both NE and SF are anything goes as far as genre and topic.
"Recently I've found the mic makes a difference." - AJOwens
mo
Marmoset
Posts: 189
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass
Recording Method: behringer umc202hd, guitar rig, cubase, garageband, audition, reason
Submitting as: duboce triangle
Location: hell a

Re: Songwriting competition judging mechanisms

Post by mo » Mon Jun 10, 2019 9:49 pm

See I basically disagree with the premise that the wild swings indicate something undesireable. I think that it proves that the judges aren’t being swayed by groupthink and that each judge has individual preferences/tastes/criteria.

In the case of something Frankie’s exit, where he got two 2nd place votes and 2 second to last place votes and got eliminated, that’s a situation that I think could stand to be reviewed. On the one hand, it’s a song that polarized the judges and therefore it is fair that it not get a top spot, but probably it should have passed, everything else being equal.

I’ve thought about it a little, and I’m sure this discussion has been had before, but I do think the judges need to all deliver a statement of their judging philosophy, musical tastes, and how they intend to approach the challenges, before the competition so that people have time to understand and digest them. If there was such a thing this time, I missed it, which could entirely be my fault, but if there wasn’t, it should be more prominent. That would help a lot, because as glennny noted, you make the songs fro the judges, not for only an audience of people who like your niche. I would hope that clearer information would make a huge difference.

The other information that would make a huge difference is knowing how the challenges are going to be evaluated. I’ve seen a little confusion sometimes as to what the letter vs the spirit of the law is, as it were, and so more specificity as to what the range is might help.

Finally in terms of actual systems, the easiest thing to do in order to prevent outcomes that feel “unjust” is simply to weight the top. In Nur Ein, after round Zero you could give a top rank a +3 point value, 2nd rank +2, 3rd rank +1, everything else as same. That has the effect of making getting a top rank more valuable because just having one top rank could give you enough extra points to stave off elimination if someone else ranked you bottom. In the situation where you got all 1s and one top rank, it probably wouldn’t save you (depending on the round, I haven’t done all the math—I’m not suggesting this exact weighting, but just giving an example). It would help in making sure a song that got 3rd and 4th ranks but no 1st ranks would have a harder time winning a round against a song that got three top ranks for example.

Just a few thoughts, hope they give someone else some ideas.
User avatar
ujnhunter
Loris
Posts: 1394
Joined: Fri Mar 07, 2008 1:09 pm
Instruments: Bass, Keyboards, Crummy Guitar & Animal Noises (especially Donkeys)
Recording Method: Reaper 5.9x, Tascam FireOne/Behringer UMC202HD/Avid Eleven Rack/Line 6 UX2, Win 7 PC / Win 10 Laptop
Submitting as: Cock, Chth*.*, D.A.H. (Der Alter Hahn)
Pronouns: His Infernal Majesty
Location: CT, USA
Contact:

Re: Songwriting competition judging mechanisms

Post by ujnhunter » Tue Jun 11, 2019 6:07 am

Who's to say that the judges that ranked you bottom aren't correct? Why should someone be saved by default and not eliminated by default?
-Ujn Hunter
Photovoltaik - Free 6 Track EP - Song Fight! Liner Notes
Billy's Little Trip wrote:I must have this....in my mouth.....now.
User avatar
glennny
Loris
Posts: 1978
Joined: Sat Mar 04, 2006 2:39 am
Instruments: Guitar, Bass, Mandolin, Dobro, Banjo, E-Bow, Glock
Recording Method: Garageband
Submitting as: Berkeley Social Scene
Location: Castro Valley, California

Re: Songwriting competition judging mechanisms

Post by glennny » Tue Jun 11, 2019 6:22 am

Finally in terms of actual systems, the easiest thing to do in order to prevent outcomes that feel “unjust” is simply to weight the top. In Nur Ein, after round Zero you could give a top rank a +3 point value, 2nd rank +2, 3rd rank +1, everything else as same......
I'd like to bet on Rain Watt to win
Max Bombast to place
and Rachel to show

the closer the songs are in quality the higher potential for wild swings in rankings. They're all good signals, the strength is in the noise.

I grew to really dislike challenges that took me away from writing a song. In my opinion there was way too much time spent on dealing with other peoples songs, rather than trying to write an original song.
Phillipso, Older Brothers, Semolina Pilchards, Zipline , Thank Glennny for the Frisbee, The Odoriferous Valley, The Worldly Self Assurance, Berkeley Social Scene, Very Gentle Knives, Daddy Bop Swing Set, GUNS, The Kraken Lives, Cavedwellers
User avatar
Pigfarmer Jr
Capuchin
Posts: 743
Joined: Sat Mar 21, 2009 6:13 am
Instruments: Guitar
Recording Method: Br-900CD and Reaper to mix
Submitting as: Pigfarmer Jr, Evil Grin, Pork Producer, Gilmore Lynette Tootle, T.C. Elliott
Location: CoMo
Contact:

Re: Songwriting competition judging mechanisms

Post by Pigfarmer Jr » Tue Jun 11, 2019 8:58 am

I think there is a difference in a song with one low ranking and the rest very high vs a song with scattered rankings or multiple high and low rankings. And honestly, the most intriguing songs are the ones that have high and low scores. I wonder how many songs I truly love that friends have hated (and vice versa.) Especially on the first listen or three.

In the example above, kicking out the high and low score would "help" the song with one low score and the rest mid to high scores. It wouldn't do anything for the song with two first place votes a middle vote and two last place votes. Which makes a lot of sense to me.

I really like the idea of SF songs being ranked as a score. But it takes longer to vote that way. Is that a good or bad thing? I'm not sure it's either. But those folks who review the songs wouldn't have *that* much more work to do to rank them. Those of us who don't always review would have to spend more time to rank them. But the added bonus is that maybe we'll get more reviews (or I'll be more consistent in posting reviews?) And it doesn't address the discrepancy in rankings that we're talking about.
Evil Grin has an album: https://evilgrin.bandcamp.com/
Spotify: https://open.spotify.com/album/1Dcqs6LhwsoYS6EDGPhCCk

My solo Album: http://tcelliott.bandcamp.com/

Ad astra per alia porci - Steinbeck
(To the stars on the wings of a pig.)
Lunkhead
Gorilla
Posts: 6069
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/reason/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead » Tue Jun 11, 2019 10:26 am

glennny wrote:
Tue Jun 11, 2019 6:22 am
Finally in terms of actual systems, the easiest thing to do in order to prevent outcomes that feel “unjust” is simply to weight the top. In Nur Ein, after round Zero you could give a top rank a +3 point value, 2nd rank +2, 3rd rank +1, everything else as same......
I'd like to bet on Rain Watt to win
Max Bombast to place
and Rachel to show
Maybe you were joking but https://en.wikipedia.org/wiki/Ranked_voting might be an interesting option.

(Also just to be clear my intended topic here is not "What's wrong with Nur Ein" "How to change Nur Ein" etc. Just brainstorming for more different ideas because I'm dreaming of trying to operate something similar to Nur Ein, on top of Song Fight, unofficially (just as myself not as a fightmaster), but with some differences in the judging mechanics.(
mo
Marmoset
Posts: 189
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass
Recording Method: behringer umc202hd, guitar rig, cubase, garageband, audition, reason
Submitting as: duboce triangle
Location: hell a

Re: Songwriting competition judging mechanisms

Post by mo » Tue Jun 11, 2019 11:00 am

Yeah I mean we have ranked voting now in Nur Ein, more or less (ie not popular but with judges), and as a voting system it functions fine, IMO. No system that involves humans is going to be objective or foolproof, I think we all agree, so given that, the system itself is mostly ok.

If you want a different system for something else, I would suggest doing some test runs of the Olympic model you guys were discussing earlier. So something like 1-10 points assigned for categories like songcraft, originality, arrangement, mixing/mastering, vocal performance, personal. Where personal can be whatever the judge personally feels. It would be a max of 60 points per voter, so if you culls, you’d need to have good tiebreakers, OR a system such that everyone who gets a certain number of points passes. If everyone is over the line, everyone passes. Then you just keep moving the line up as rounds go on. So you could make it cumulative, so your score in the previous rounds matters beyond just advancement.then the highest points at the end of however many rounds wins. Just throwing it out there.
User avatar
Pigfarmer Jr
Capuchin
Posts: 743
Joined: Sat Mar 21, 2009 6:13 am
Instruments: Guitar
Recording Method: Br-900CD and Reaper to mix
Submitting as: Pigfarmer Jr, Evil Grin, Pork Producer, Gilmore Lynette Tootle, T.C. Elliott
Location: CoMo
Contact:

Re: Songwriting competition judging mechanisms

Post by Pigfarmer Jr » Tue Jun 11, 2019 11:29 am

For a nur ein type of competition, the idea of cumulative voting is appealing, but maybe more in the idea than in practice (possibly.)
Evil Grin has an album: https://evilgrin.bandcamp.com/
Spotify: https://open.spotify.com/album/1Dcqs6LhwsoYS6EDGPhCCk

My solo Album: http://tcelliott.bandcamp.com/

Ad astra per alia porci - Steinbeck
(To the stars on the wings of a pig.)
Lunkhead
Gorilla
Posts: 6069
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/reason/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead » Tue Jun 11, 2019 2:50 pm

mo wrote:
Tue Jun 11, 2019 11:00 am
Yeah I mean we have ranked voting now in Nur Ein, more or less (ie not popular but with judges), and as a voting system it functions fine, IMO. No system that involves humans is going to be objective or foolproof, I think we all agree, so given that, the system itself is mostly ok.
The judges rank every entry but then the rankings are just summed. We don't have a ranked choice voting system where an entry that gets a majority of first place votes would win, or if no entry got a majority of first place votes, then second-place rankings would be considered, etc.

https://ballotpedia.org/Ranked-choice_voting_(RCV)
https://en.wikipedia.org/wiki/Instant-runoff_voting

That could be complicated to tally up but there are tools out there: https://petertheone.github.io/IRV/ and it could maybe be applied "in reverse" to determine who gets eliminated?
Lunkhead
Gorilla
Posts: 6069
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/reason/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Songwriting competition judging mechanisms

Post by Lunkhead » Tue Jun 11, 2019 2:51 pm

OR a system such that everyone who gets a certain number of points passes. If everyone is over the line, everyone passes. Then you just keep moving the line up as rounds go on. So you could make it cumulative, so your score in the previous rounds matters beyond just advancement.then the highest points at the end of however many rounds wins. Just throwing it out there.
Those are definitely interesting ideas! I think SpinTunes maybe experimented with that idea, of folks continuing to participate rather than being eliminated, with the final results being cumulative across all the rounds. Spin or glennny, did that happen?
mo
Marmoset
Posts: 189
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass
Recording Method: behringer umc202hd, guitar rig, cubase, garageband, audition, reason
Submitting as: duboce triangle
Location: hell a

Re: Songwriting competition judging mechanisms

Post by mo » Tue Jun 11, 2019 3:28 pm

Oh, ranked choice voting. Yeah, that tends to work better when you have lots of votes, so if you're planning to do something with a popular vote instead of a limited judging pool, that could be a great way to go.
mo
Marmoset
Posts: 189
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass
Recording Method: behringer umc202hd, guitar rig, cubase, garageband, audition, reason
Submitting as: duboce triangle
Location: hell a

Re: Songwriting competition judging mechanisms

Post by mo » Tue Jun 11, 2019 3:50 pm

Another suggestion with the points categories is that of course categories can be worth different amounts of points, so let's say songcraft and originality are worth 10 points maximum each, but maybe mixing/mastering is worth 5 maximum, etc. My big concern with this system is that in any one round, you're likely to have a whole bunch of ties, or the possibility that every song is pretty good, which is why I suggested just having a pass/fail line. The reason for increasing the threshold as rounds go on is to make it more difficult, so that only songs of a certain quality will get through--but of course that is still the totality of the submitted song. Maybe for the final heats such as semi-final and final, you could then also add in ranked choice voting from judges so that there's a clear mechanism for winning/losing. Maybe?
User avatar
fluffy
Baboon
Posts: 9696
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: they/them||she/her
Location: The Plaidlands (also, Seattle)
Contact:

Re: Songwriting competition judging mechanisms

Post by fluffy » Tue Jun 11, 2019 5:09 pm

Okay so back to the question about voting systems: I'm a fan of marking things as "good" or "bad" and letting the overall ratio determine a winner. Mathematically it's similar to what we have on Song Fight! right now but the psychology changes the equation a bit. Maybe each vote could also have a +1 bonus for "favorite song" or something. CoverFight! back in the day had a voting system sorta like that except the "favorite" and "good/bad" were separated which caused some really weird scenarios (like that time that my entry had the highest rating but 0 votes because everyone liked it but it was nobody's favorite).

Ranked-choice/instant-runoff voting is great if you want to spend a lot of time deciding your exact order of preference but that can be difficult in a context where you have a lot of different things which are good or bad on different merits.

Once upon a time I was thinking of ways of doing mini-bracket-style voting, where people are given random pairings of things and asked which one is better between the two, and then trying to figure out the overall sort order based on which ones win in which pairings most often, although that ends up causing all sorts of non-transitive properties that make an objective answer difficult to come by (the classic Rock-Paper-Scissors situation).

Back in the day, GarageBand (the website, not the Apple product) used random-pairings comparisons to handle their challenge voting, although I have no idea how useful it was in the end or what the underlying final ordering was. There's probably a reason nobody remembers GarageBand.
User avatar
ken
Lemur
Posts: 3415
Joined: Sat Sep 25, 2004 6:10 pm
Instruments: Guitar, bass, drums, keys
Recording Method: MOTU 828x, Cubase 10
Submitting as: Ken's Super Duper Band 'n Stuff
Pronouns: he/him
Location: oakland, ca
Contact:

Re: Songwriting competition judging mechanisms

Post by ken » Wed Jun 12, 2019 9:04 am

fluffy wrote:
Tue Jun 11, 2019 5:09 pm
nobody remembers GarageBand.
Garageband was great because if you did a certain number of reviews you got a gift card to buy a CD of your choosing from some Canadian record store I can remember. That was totally worthwhile.
Ken's Super Duper Band 'n Stuff - Berkeley Social Scene - Tiny Robots - Seamus Collective - Semolina Pilchards - Cutie Pies - Explino! - Bravo Bros. - 2 from 14 - and more!

i would just like to remind everyone that Ken eats kittens - blue lang
mo
Marmoset
Posts: 189
Joined: Fri Oct 08, 2004 9:46 pm
Instruments: guitar, bass
Recording Method: behringer umc202hd, guitar rig, cubase, garageband, audition, reason
Submitting as: duboce triangle
Location: hell a

Re: Songwriting competition judging mechanisms

Post by mo » Wed Jun 12, 2019 12:16 pm

ken wrote:
Wed Jun 12, 2019 9:04 am
fluffy wrote:
Tue Jun 11, 2019 5:09 pm
nobody remembers GarageBand.
Garageband was great because if you did a certain number of reviews you got a gift card to buy a CD of your choosing from some Canadian record store I can remember. That was totally worthwhile.
What's a CD?
User avatar
BoffoYux
Tamarin
Posts: 445
Joined: Tue Jan 11, 2011 9:22 pm
Instruments: Keys, Clunking, SFX and Strings
Recording Method: Audacity, Adobe, and other 'A' titled software
Submitting as: Boffo Yux Dudes
Location: New England
Contact:

Re: Songwriting competition judging mechanisms

Post by BoffoYux » Wed Jun 12, 2019 12:38 pm

Chumpy wrote:
Thu Jun 06, 2019 7:49 pm
I wouldn't be surprised if BYD lost points in some judge's estimation due to their quiet mixes.
THAT'S the reason why we lost points! I thought it was our acid wit and rogueish good looks...

(However, we will start to tweak the mixes further next time. Thanks for the thoughts on what we've been ignoring all these years.)
Post Reply