songfight.org Homepage Issues

Let us know when something isn't working correctly, or if you find a typo. Do not post complaints or suggestions here.
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

songfight.org Homepage Issues

Post by Lunkhead »

Maybe we can collect all these random little things in one thread rather than starting a new thread for each?

First off, while I am happy to keep seeing Merisan's only victory touted on the homepage, it made me realize that the "five years ago" and "ten years ago" content has gotten stale.
User avatar
Manhattan Glutton
Ice Cream Man
Posts: 1530
Joined: Tue Feb 15, 2005 12:10 pm
Instruments: Angst
Recording Method: REAPER
Location: Madison, WI
Contact:

Re: songfight.org Homepage Issues

Post by Manhattan Glutton »

The site started using mod_pagespeed - which apparently embeds images as base64 into the HTML. I had to fix any scraping I was doing.

Not that I'm complaining. I fixed it.
If I had a dollar for every one of my songs j$ has called a 90s pastiche, I'd have $1 for every song I've written.

Nur Ein Archives | The New Ugly Podcast
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: songfight.org Homepage Issues

Post by Lunkhead »

Whoa, mod_pagespeed and embedded images? That's kind of some futuristic stuff for Song Fight! How did that get there...?
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: songfight.org Homepage Issues

Post by fluffy »

Automatically.
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: songfight.org Homepage Issues

Post by jb »

I turned on the auto-google-analytics for the domain, and was required to turn on pagespeed. Sorry f I messed anything up in your scripts.

J
blippity blop ya don’t stop heyyyyyyyyy
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: songfight.org Homepage Issues

Post by fluffy »

mod_pagespeed is pretty cool, although it seems like a greedy optimization that ignores that a lot of the small images are going to be cached (but then again, the amount of data embedded via base64 is probably less than the total amount of data and latency incurred by added HTTP transactions), and it also is a bit aggressive at "optimizing" Javascript in ways which break Project Wonderful's ad-checking bot (not that that affects Song Fight, but it does affect me). It definitely will break anything that scrapes HTML, though (and the way it breaks will always be a moving target), but HTML-scraping is also a terrible thing to do.

Since the Song Fight jukebox uses the underlying archive data for its updates, it's probably better for those of us who are scraping the site to switch to using Lunkhead's jukebox API for data updates instead. It'd definitely be safer and more reliable, in any case.
User avatar
Manhattan Glutton
Ice Cream Man
Posts: 1530
Joined: Tue Feb 15, 2005 12:10 pm
Instruments: Angst
Recording Method: REAPER
Location: Madison, WI
Contact:

Re: songfight.org Homepage Issues

Post by Manhattan Glutton »

The scrape that broke was the news page (eh, since news.rss is broken/not-robust or whatever). I just had to change the user agent and the XPath to be less specific. Not that it mattered for anybody. I swear I'll get to releasing this app this year.
If I had a dollar for every one of my songs j$ has called a 90s pastiche, I'd have $1 for every song I've written.

Nur Ein Archives | The New Ugly Podcast
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: songfight.org Homepage Issues

Post by Lunkhead »

I could poll the news.rss and import/archive the news items and make them available as json/xml/etc. if anybody wanted to use that.
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: songfight.org Homepage Issues

Post by fluffy »

rss.php's output should be as robust as any other XML document, but you need to parse it as XML, not using regular expressions or whatever. Although I have no idea what mod_pagespeed could have done to it anyway (as far as I can tell, mod_pagespeed isn't affecting it at all though, based on a diff of its raw output vs. curl'ing it).

Or is there some other RSS feed that you're referring to? There isn't one named news.rss so far as I'm aware.
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: songfight.org Homepage Issues

Post by Lunkhead »

Oh, right, the correct name is "rss.php" it looks like.

Just offering the news items in a different format isn't really adding much value, but I was thinking that archiving them and making them searchable might be kinda neat.
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: songfight.org Homepage Issues

Post by fluffy »

The value it adds is that you can subscribe to it via a feed reader so that you know when a new title is posted, when the new fight is posted, and when a new news item has been posted. You know, like any other RSS feed.
User avatar
Manhattan Glutton
Ice Cream Man
Posts: 1530
Joined: Tue Feb 15, 2005 12:10 pm
Instruments: Angst
Recording Method: REAPER
Location: Madison, WI
Contact:

Re: songfight.org Homepage Issues

Post by Manhattan Glutton »

I reported an issue with rss.php about a year ago, and out of that additional issues cropped up with regards to its existence and maintenance, so I figured it was a lost cause. If that's changed, I certainly would like to know, but I've already gone to the trouble to make my own news scrape: http://api.sfbase.net/news.php
If I had a dollar for every one of my songs j$ has called a 90s pastiche, I'd have $1 for every song I've written.

Nur Ein Archives | The New Ugly Podcast
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: songfight.org Homepage Issues

Post by fluffy »

What was the problem? Has it been fixed? It's almost certainly going to be more reliable than scraping HTML.
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: songfight.org Homepage Issues

Post by Lunkhead »

fluffy wrote:The value it adds is that you can subscribe to it via a feed reader so that you know when a new title is posted, when the new fight is posted, and when a new news item has been posted. You know, like any other RSS feed.
I meant the value added by me importing the data into the Jukebox via polling the RSS.
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: songfight.org Homepage Issues

Post by fluffy »

Oh, that. Yeah, I don't know, might be neat, but since all of the item types are undifferentiated there'll be a lot of information that isn't really useful. I suppose I could add tags to indicate whether it's news, title, songs posted, or results, though. It'd be pretty easy since all the underlying data comes from different sources anyway.

[edit] added
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: songfight.org Homepage Issues

Post by jb »

I've always been frustrated by the fact that we can't lump all the songs for a fight into one RSS post with multiple attachments. So in my RSS feed I get 20 posts, one for each entry. Somewhat cumbersome for the readers that I use.

Also, Lunkhead it would definitely be useful to have the titles as an RSS feed. I'd like to be able to create an IFTTT that posts the new title to the G+ community or possibly to the SF FB page. You have the @sfarchivist Tweeting the new titles, but IFTTT can't do Twitter any more. :P
blippity blop ya don’t stop heyyyyyyyyy
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: songfight.org Homepage Issues

Post by jb »

I am also frustrated that I can't easily upgrade these forums, and that I am nervous that my carefully hacked-in review functions will need to be carefully re-hacked if I upgrade the thing anyway.

This is why I hate programming.
blippity blop ya don’t stop heyyyyyyyyy
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: songfight.org Homepage Issues

Post by Lunkhead »

jb wrote:Also, Lunkhead it would definitely be useful to have the titles as an RSS feed. I'd like to be able to create an IFTTT that posts the new title to the G+ community or possibly to the SF FB page. You have the @sfarchivist Tweeting the new titles, but IFTTT can't do Twitter any more. :P
"IFTTT can't do Twitter any more" :roll: That is super lame. I think I already have a JSON endpoint for that data now so adding RSS formatting should be pretty trivial. Sadly the Jukebox and all my sites/email/IM are down at the moment...
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: songfight.org Homepage Issues

Post by Lunkhead »

jb wrote:I am also frustrated that I can't easily upgrade these forums, and that I am nervous that my carefully hacked-in review functions will need to be carefully re-hacked if I upgrade the thing anyway.

This is why I hate programming.
That's not really programming's fault, that's PHPBB's fault. :P
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: songfight.org Homepage Issues

Post by fluffy »

jb wrote:I've always been frustrated by the fact that we can't lump all the songs for a fight into one RSS post with multiple attachments. So in my RSS feed I get 20 posts, one for each entry. Somewhat cumbersome for the readers that I use.
That's mostly a failing in how podcasts were "designed," i.e. they're just RSS with a single media attachment. There's nothing about RSS itself that prevents you from using multiple enclosures - it's just that none of the podcast subscription engines that people use (notably iTunes) support it.

http://songfight.org/rss.php doesn't bother trying to do enclosures for entries; it just states as a news item when the songs are available. (Or more accurately, it produces a news item for what the current title is, so it's always regenerating the same item for as long as the title doesn't change.) If anyone wants to get the individual pieces of content they can always use the podcast feed, which is unaffected.

I could actually change rss.php to provide the songs as enclosures on the "now playing" item but I don't know what value that would add, aside from making it slightly easier for people to try to listen to songs from their enclosure-savvy RSS reader (which will probably get screwed up by the anti-hotlink thing if it's web-based).
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: songfight.org Homepage Issues

Post by jb »

fluffy wrote:http://songfight.org/rss.php doesn't bother trying to do enclosures for entries; it just states as a news item when the songs are available. (Or more accurately, it produces a news item for what the current title is, so it's always regenerating the same item for as long as the title doesn't change.) If anyone wants to get the individual pieces of content they can always use the podcast feed, which is unaffected.
IIRC it does that because when we were figuring out how to do it we discovered that you couldn't do multiple enclosures well, so we went with the simplest solution.

For a while I was playing around with trying to write a script that would take all the entries and put them together in one MP3, with computer-generated voice between each song announcing the band name. Never really got traction on it, though it seemed like it should be possible based on some of the libraries that are available...
blippity blop ya don’t stop heyyyyyyyyy
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: songfight.org Homepage Issues

Post by fluffy »

right, I meant it was iTunes et al which don't support multiple enclosures well, not RSS itself.

Turning it into a single long-form mp3 with voiceovers in between wouldn't be hard. sox can join together multiple wav files easily, and there's probably a more clever way of joining the mp3 frames directly without having to reencode (apparently you can just cat *.mp3 together and it'll Just Work in most players but I'd worry about intra-file id3 tags screwing things up for some).
Post Reply