Backup Planning - what's your scheme?

Ask questions and get answers about how to make music in any particular way. Hardware or songwriting or whatever.
Post Reply
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Backup Planning - what's your scheme?

Post by jb »

I recently had a minor loss of data as part of moving to a new laptop. I can tell you the whole story if you want, but bottom line is it caused me to try to figure out some new arrangement for backing things up. Here's where I stand:
  1. I have a 10TB Degoo account I got some time ago. I'm backing it all up to that, just everything, set to "do not delete if i delete from my computer". Eventually that will have just a giant pile of stuff on it, but it's paid for and was a "Life Time Deal" so I might as well use it.
  2. I still have an external drive and will be using Time Machine (I use Mac) to backup regularly. Time Machine isn't perfect, but it's at least a low-impact backup service that happens automatically. I need to figure out how to get it to back up via wifi rather than having the drive attached to the laptop.
  3. I upgraded to Dropbox Plus (2TB of storage) and have aliased my Logic folder into my Dropbox folder. That will be my everyday contingency plan.
So, all my music data will be:
  • On my laptop drive
  • In Degoo
  • On Dropbox
  • In Time Machine
If that doesn't keep me from losing shit, I just plain give up.

QUESTION: Anybody see any better way of doing this? What's your backup plan?
blippity blop ya don’t stop heyyyyyyyyy
User avatar
jast
Ice Cream Man
Posts: 1325
Joined: Tue Jul 29, 2008 7:03 pm
Instruments: Vocals, guitar
Recording Method: Cubase, Steinberg UR44
Submitting as: Jan Krueger
Pronouns: .
Location: near Aachen, Germany
Contact:

Re: Backup Planning - what's your scheme?

Post by jast »

I've read multiple accounts from random people on the internet claiming that Degoo mysteriously lost some of the data (or compressed? Who knows) and wouldn't let them download some of the data. At least one person also claimed downloads were insanely slow for them. Just something to consider. That aside, your plan looks good to me. Just remember to periodically check that all your backups are still functional. :)

I have a NAS/media center which stores copies of all of my optical media, mostly for convenience. Beyond that:
- I backup all of my own data on a locally connected 8 TB disk, using Borg (which is slower than Time Machine, which I don't have because I don't use Apple stuff, but it's also a lot more space efficient when it comes to storing changed files).
- Some of my most critical data is stored encrypted in Dropbox and similar services.
- I've been thinking about storing encrypted copies of a "fairly important" subset of my data (including music and such) in an object store like Backblaze's B2 (which is just about the cheapest such thing that you could find, except maybe Online's C14 which is super cheap for long term storage but a little more complicated) in addition to the local backup. So far I just haven't felt like putting in the work...
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Backup Planning - what's your scheme?

Post by fluffy »

I keep active/recent projects on my desktop, which is backed up via Time Machine to both a USB-attached 3TB hard drive and a Synology NAS (DS414) over the network, and also to Backblaze B2 with the Arq backup client.

Archived projects live on the Synology NAS, which gets backed up to Backblaze B2 as well (using Synology Cloud Sync, which unfortunately doesn't encrypt files like Arq does so I need to make sure my buckets stay private).

The NAS itself is a 4-drive RAID-5 and can tolerate one drive failure, and I'm using a mix of Seagate Barracuda Compute and WD Red NAS drives, and have made sure to stagger their install times to reduce the possibility that two fail at once. The Synology is also incredibly aggressive about going into read-only mode the moment there's any indication of disk trouble. I do wish I had a better failure mitigation strategy for my NAS (i.e. something on-site for recovery), but the only option I can really see for that is to get a second NAS that acts as a backup target and that takes a lot more power and space than I want to put into it, and at that point I'd rather get a bigger array that would allow me two hot spares instead of just one (although that's starting to feel like overkill even for me).

Oh and for what it's worth, I also have multiple backups of the songfight.org site code and its git repository, and both my desktop and NAS have a reasonably up-to-date mirror of the music archive (albeit with the id3 tags rewritten to my preference). Y'know, just in case something happens to the site, either by Dreamhost hecking up or by it getting hacked again.
User avatar
ken
Hot for Teacher
Posts: 3869
Joined: Sat Sep 25, 2004 6:10 pm
Instruments: Guitar, bass, drums, keys
Recording Method: MOTU 828x, Cubase 10
Submitting as: Ken's Super Duper Band 'n Stuff
Pronouns: he/him
Location: oakland, ca
Contact:

Re: Backup Planning - what's your scheme?

Post by ken »

I tend to just lose everything when I get a new computer. Not the greatest plan, but I haven't had too many issues with it yet. I do have a pile of CDRs, hard drives and old computers lying around as well as VCR tapes, cassettes and DATs. I only recently had a couple of DATS and cassettes converted to more accessible file formats, but haven't done anything with those yet either. I guess I'm saying I'm just not that precious with my music.

Really what I'm saying is THANK YOU SONGFIGHT FOR KEEPING MY MUSIC ALIVE ON THE INTERNET FOR SO LONG!!!

(also I miss songhole.org)
Ken's Super Duper Band 'n Stuff - Berkeley Social Scene - Tiny Robots - Seamus Collective - Semolina Pilchards - Cutie Pies - Explino! - Bravo Bros. - 2 from 14 - and more!

i would just like to remind everyone that Ken eats kittens - blue lang
User avatar
irwin
Mean Street
Posts: 617
Joined: Mon Jan 22, 2007 10:09 pm
Instruments: insecurity
Recording Method: scotch
Submitting as: suckweasel, kasper, nutwalls
Location: Woodland, CA

Re: Backup Planning - what's your scheme?

Post by irwin »

I currently do time machine to an external USB drive, but I don’t like it. I’ve had way too many drive failures. I tried to do time machine over iSCSI to a freebsd machine, but every (I tried a few) OSX initiator was… less than reliable. There must be a better answer, c’mon, it’s 2020, people. Flying cars, jetpacks, working cross-platform distributed filesystems, is that too much to ask? (Seriously, almost 40 years later and AFS is the best we’ve done?)
"Ouch. I wonder if this guy sounds like this when he speaks." -- Puce
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Backup Planning - what's your scheme?

Post by fluffy »

I miss AFS although that's mostly misplaced nostalgia for a thing I never actually used. But it always seemed neat as heck.

CAP theorem is a pretty decent take on why distributed filesystems are hard to do well, and most of the tradeoffs made for the C and P part severely impact the A part, which is unfortunately murder on desktop computing systems (since apps are never written with that in mind). Partial sync-based things like dropbox/nextcloud/etc. kinda pave over it but it can still cause problems.

I do wish more pooled filesystems embraced object-level rather than block-level redundancy though. It makes consistency checking and restoring a lot easier. WinFS had some great ideas in that regard.

ZFS is pretty nice. Shame Apple gave up on trying to get it to work with macOS.
User avatar
irwin
Mean Street
Posts: 617
Joined: Mon Jan 22, 2007 10:09 pm
Instruments: insecurity
Recording Method: scotch
Submitting as: suckweasel, kasper, nutwalls
Location: Woodland, CA

Re: Backup Planning - what's your scheme?

Post by irwin »

I always thought it might be neat to dump my projects into git lfs. I often want to start projects on my laptop and continue on my desktop or vice-versa, and it seems like that would be a lot easier to keep track of than my current technique of “rsync shitcircus”. But then again in the absence of a plugin to assist, it would require discipline on my part, and given how often my laziness hoses repositories in the job I actually get paid to do, I am not optimistic.
"Ouch. I wonder if this guy sounds like this when he speaks." -- Puce
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Backup Planning - what's your scheme?

Post by fluffy »

Every time I look into setting up an LFS repository I end up remembering why I never do so. It's a lot of hassle and every configuration I care about is "unsupported."
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: Backup Planning - what's your scheme?

Post by jb »

Just found a thing that I thought would be supported by everything but seems to be supported by almost none of the things I currently own: Networked drives.

Me: "I should put this drive on the network to Time Machine rather than having to hook it up to my laptop constantly"
Me: "I can probably just plug it into my Orbi and it'll Just Work."
Orbi: "HA, NOT LIKELY"
Internet: "Yeah that Orbi USB hole actually doesn't do anything at all. It's for the future."
Me: "Which future?"
Internet: "Not this one."
Me: ...
Me: "Well darn, ok how about I plug in this Airport Express I have extra, surely that'll just put my drive on the network like kapow."
Airport Express: "NO."
Me: "God damn it."

Anybody know of a device that will let you attach your existing, USB, external drives to your network? That isn't like, trying to be a whole enclosure and shit?
blippity blop ya don’t stop heyyyyyyyyy
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Backup Planning - what's your scheme?

Post by fluffy »

Just putting it on the network isn't enough, it needs to specifically support the Time Machine protocol (which is an extension to SMB). Synology does (with some fiddling to get it working), and I think it'll also let you attach existing USB drives as a Time Machine volume although I haven't tried that.

https://support.apple.com/en-us/HT202784 claims that you should be able to use a USB drive on an Airport Extreme or Time Capsule, as long as it's formatted correctly. And they say nothing about Airport Express so I'd assume that's a big fat "no."
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: Backup Planning - what's your scheme?

Post by jb »

Christ I’m done with fiddling. And yeah the internet says expresses don’t support it.
blippity blop ya don’t stop heyyyyyyyyy
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Backup Planning - what's your scheme?

Post by fluffy »

Check your local thrift stores for used Airport Extremes. Sometimes you can get them for stupid cheap there. Like, a year ago I saw three of them for $10 each, and one turned out to be a 3TB Time Capsule. (Then I ended up reselling them on eBay for like $200 total.)

I don't recommend them as access points anymore but they're still great as attachment points for USB drives and printers. And, hey, $10.
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: Backup Planning - what's your scheme?

Post by jb »

I actually had an extreme. This is what I get for throwing things away, MARIE

THAT AIRPORT WXPRESS WIULD SOARK A LOT OF JOY RIGHT ABOUT NOW
blippity blop ya don’t stop heyyyyyyyyy
Post Reply