Muxtape

Online

We’re back, although The Planet is still without power and might be for a while. Peter just moved the entire site to EC2, no big deal ;) Thanks for your patience.

Still down

According to The Planet (our hosting company), “[Yesterday] evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site.” No ETA so far, it’s about noon right now.

Muxtape is down as of 3:01 am because of a problem at our host. They lost power to a data center, and that’s pretty much all I know right now. I don’t anticipate any data loss, but I have a backup of everything as of 2:07 this afternoon just in case. We’ll be back as soon as the power comes back on.

Back Online / Debriefing

On May 21st, Muxtape experienced an error that caused every song in the database to be overwritten with the wrong artist & title. I initially reported this as a malicious act, but after taking the site down and investigating further it was clear that it was a fault of the software, for which I am wholly responsible. The last available complete backup of the songs database was three weeks old; so while the site is now repaired and back online, most songs uploaded in the last three weeks have disappeared.

I’m truly sorry for this massive inconvenience. Data integrity is something I take seriously, and to prevent this from happening again we’ve started creating complete backups of the database twice a day. In addition, Muxtape will soon be joined by two more developers (as opposed to one developer who’s constantly distracted by everything else it takes to start and run a business/web site).

Thanks for your patience and understanding through this, and please contact us if you have any questions or concerns.

Update #3

We’ve narrowed down the issue, but we’re triple-checking everything to make sure it doesn’t happen again before we bring the site back up. I’m very sorry to report that, contrary to what I said earlier, there’s going to be some lost data. I won’t drag out all the details, but Muxtape has two backup methods and they both failed in a particular way that screwed the other out of being useful. A perfect storm, if you will. I take full responsibility; there was a typo in the code a few weeks ago that I swore up and down had been fixed and it would’ve rendered this whole thing moot, but it somehow didn’t get saved and went unnoticed until now. Extremely frustrating.

What it boils down to is that only 15% or so of songs uploaded in the last three weeks (and 100% of songs before that point) are going to be restored. I’m very sorry about this, and I’m working with some brilliant minds to make sure it never happens again. Muxtape will be back online this afternoon.

Update #2

We’re hard at work restoring the database, which is a little more involved than it would be otherwise thanks to various mitigating circumstances. Once the site comes back up, every single muxtape should be restored with all their songs in the right order. However, some songs uploaded in the last three weeks that have a custom title may need to be relabeled. We’re very sorry for the inconvenience, thanks for being patient during our downtime.

Update

I don’t want to bring the site up again until I’m sure what happened this afternoon won’t get repeated, but there are backups of the overwritten data and I anticipate most of it being restored successfully.

Muxtape temporarily down

Update: what initially appeared as someone gaining access to our server could have possibly been an accident. We are still investigating the cause of the problem. I’ve taken the site down temporarily as a precaution. Thanks for your understanding and patience.

The top two feature requests right now are M4A/AAC support and scrobbling for Last.fm. Both are coming, but there’s a couple smaller things I’d like to implement first (which you’ll be seeing soon.) AAC support is going to take a little while because it *just* got added into Flash and the way it deals with those files is much different than regular MP3s, and the library I’m using now doesn’t support it at all. That said, it is coming!