This week Chris and Larry discuss the Ma.gnolia’s data loss, what is has meant for the service and in the community, and what may be coming in the future.
You can watch below, or if you’d rather listen, you can download the mp3.
Citizen Garden Episode 11: Whither Ma.gnolia? from Larry Halff on Vimeo.
34 Comments
Thanks, Larry and Chris! Larry, this has to be so hard, and I appreciate your honesty and making this as transparent as possible, especially while you are still trying to figure out exactly what went wrong. It demonstrates your integrity and I think will go a long way towards re-building trust with the Ma.gnolia user community. I really appreciated your thoughts towards the end of the conversation about how we use data in the moment, and I think there are lessons that we can all carry forward in terms of how we use store data in the cloud.
Mostly, I am thrilled to know that Ma.gnolia will be coming back. I really miss it! I have been using Delicious as a back up (and this is certainly giving me the opportunity to use and compare the two tools, and as a good info professional, I should have done this a long time ago, anyway). But it is just not the same. I miss the community there. I miss the folks I was following, I miss the offers of thanks, and I miss the groups. I really look forward to the fantastic comeback.
Thanks for all that you are doing!
This was a great, and very open discussion. Obviously there are lessons to learn from the experience with Ma.gnolia, and your discussion on the cloud in general is quite important. I would love to see what could come of your discussion relating to bookmarks. Recommendations would be an awesome use of stored bookmarks. I always thought that “future” bookmarking would be super cool. For example, Disney has their Earth show coming up in April. I would love to store a link to the shows home page today, and then get reminded about it in April when the show is going to air. Ma.gnolia had some integration with Upcoming at one point that stored event detail, but I don’t think it ever really took off.
I am one of the people who moved away from Ma.gnolia in late 2006 due to some of the performance issues, but I always kept coming back. I kept bookmarks flowing into Ma.gnolia through Diigo as I used that tool. I would certainly love to see Ma.gnolia reborn with a fresh outlook on the web. I look back on the days when I one of my start pages was the Ma.gnolia chat room on Pibb with great fondness. I would love to see that community rebuilt.
Transcript: http://ratafia.info/post/78915439/transcript-and-commentary-for-whither-magnolia
Great interview, and thanks for the honesty Larry.
As Chris mentioned we, the internet at large, have no clue how the services we use are run. I’m really excited that Ma.gnolia will be returning and one thing I would ask is a reminder to download a back-up of anything I put into it. Thinking about it, this would be great for any of the cloud services I use.
As I just found this podcast, I’m looking forward to future shows.
Xserve.. Use a real server. on a real file system, that has been though real world testing.
myslq? LOL what a crappy database.
dumbasses.
So, it sounds like he never actually did a true backup. He just synch’ed data to a remote machine. Meaning that any corruption is synch’ed automatically so that both sides are corrupted. In short, he’s a dumbass.
Redundancy != Backup
I’m sorry you are a dumbass and never once looked into the proper way to backup SQL. After I heard you were running a Mac, it’s understandable. I can’t really expected you to actually know what you are doing.
* When I heard you were running a software RAID, I laughed even more. You own overpriced machines and are getting poor performance.
The only fools who mismanaged their data are Magnolia’s customers. Pure and simple, don’t put your data on hosts you don’t control. Don’t assume a free service is competently managed. Get an SLA and PAY for your hosting and publishing.
Back the standards and not the ‘startups’. How many times does this have to happen before people stop believing the hype?
Citizenship means being a competent contributor not moving from fashionable address to fashionable address so you can be ‘cool’.
This is an honest mistake by somebody who doesn’t have the expertise in an area they are responsible for but wasn’t the essence of their invention. It is *inherent* in being a “Web 2.0″ startup chasing an ideal as if this is all something new rather than lipstick on a pig because the barrier to entry for The Web is lower than it was for gopher, usenet etc. Don’t be angry at them because you were too naive not to use the service in the first place.
Hint: Apple. Google, Microsoft aren’t immune either.
A production suggestion: Seated people look much better when the camera is slightly higher than they are. Look at any newscast for examples.
Thanks Clarence, this was the first episode we filmed, so that’s good advice.
As for our Slashdot friends, welcome! We appreciate your snark and insight!
“mwmwmw
Posted February 20, 2009 at 9:34 am | Permalink
Xserve.. Use a real server. on a real file system, that has been though real world testing.
myslq? LOL what a crappy database.
dumbasses.”
Awesome feedback! Way to add constructively to the conversation.
You automatically loose 20 IQ points for spelling mistakes on a site that spell checks for you.
I do agree with you on one thing though. Using Xserve? Why?
“You automatically loose 20 IQ points for spelling mistakes on a site that spell checks for you.”
And you lose 50 points by spelling “lose” with two o’s.
So did you guys watch the Journalspace meltdown and just not do anything?
Figure out what your skill set is and hire around that. This problem could have been avoided by spending about 10 hours with a knowledgeable mysql admin.
It strikes me that database systems have no back up system built in, just as a new PC has no back up system built in.
There is no standard “configuration option”, and so a web site operator needs to choose and set up some system just as every PC user. And after one has chosen a system and is using it, one needs to make sure it works.
On the other hand, mysql or MS Windows might have been developed from the very beginning with built-in and integrated back up options, tested and proven. The way it is now back-up is an after-thought. I think this is the heart of the problem.
Oh well,
Stephan
Come on everyone. People make mistakes and they don’t know what they don’t know. The worst pilots are doctors, lawyers and programmers BECAUSE they think they know everything they need to know or can learn the rest on the way. Nobody likes to ask for help (or directions).
System backups are worthless unless you verify they can be used to restore on a different system. Have you verified the restoration of your system backups …. recently?
Backups without versioning are almost worthless. Suppose you get hacked and don’t notice the back door for a few weeks. Will your current recovery method allow you to figure out what changed and correct it?
Backups without a restoration window is “hoping” everything will be alright. “Hope is not a plan.”
Whoever said that MySQL isn’t a good database should be ignored. OTOH, if any database doesn’t do what you need it to, then find one that does. I don’t care if it is SQLite, text files, gdbm, Oracle or DB2 running on MVS.
Hire a systems/enterprise architect if you need help with this stuff. There are many cost effective ways to be certain these things “just work” without too much human intervention. A 2 hour consult is pretty cheap compared to a failed business.
The real problem is they never tested a full restore.
LOL I love how he started laughing at the beginning … and then said it wasn’t funny for some people!
I like his style!
This is a perfect example of knowing one thing well and assuming that knowledge translates into related fields. Backups left untested? Software raid? Wow, talk about some poor system administration. This could have been avoided with a few hours of research on the weekends. I will not be using them anytime soon.
“* When I heard you were running a software RAID, I laughed even more. You own overpriced machines and are getting poor performance.”
What makes you think it’s got poor performance? You don’t really understand software RAID. Many sites use Linux software RAID very successfully, and in fact it is preferred over hardware RAID, as it can be damn hard to get a replacement controller ten years down the line when it fails.
On the database side, it doesn’t surprise me that many social sites use MySQL, but it’s not exactly production stuff for this kind of usage… more of a datastore for quick,small web sites, unless of course you are called Google and know how to engineer MySQL to fulfill your needs, but it’s quite obvious thay these guys did not focus that much on the hardware behind their site. If cost was an issue they should have gone with Postgres instead of MySQL, but heck, with that hardware, they could have gone with OracleXE and not hit any Oracle limits, which would have been a much better choice.
Mirroring your data doesn’t help you against data corruption, as few guys already pointed out. Backup is serious business. Properly designing your app and properly scaling your system is serious business. To do things Ma.gnolia way is pretty dumb; without any serious planning, disasters are just waiting to happen.
Choosing MySQL as a database is ok, but certain optimization needs to be done and proper backup plan needs to be in place.
Thank you for your honesty. I have been very sad at this loss, and was hoping for a restore. I didn’t understand how there wasn’t a backup, and there wasn’t. However, I don’t blame you for what I lost, it was my information, and if it was important I should have had a backup. I loved how Ma.Gnolia was set-up, and nothing compares to it. Thank you for giving me that, and I do wish there had been a better backup systems, and we can all learn from this. Thank you for sharing. I tell everyone, computers are computers, and sometimes sh*t happens, and nobody ever understands why.
Hi guys,
I really appreciate your struggle… hindsight vision is 20/20, so ignore the moaners above. I’m sure that this has been a rough patch, but you’ll get through it and be better off in the long run for it.
On a purely selfish note though, is there a chance of a new podcast/video? Even if the subject of Ma.gnolia isn’t discussed we would sure like to hear your thoughts on the industry/status quo.
Thanks!
Hi Larry and Chris,
You are so honest ,first of all. Data loss is a great problem but you explain this case.
I look forward to your comeback.
Success with you…
Good luck.
This does sound good (as analysis), but unfortunately there are still startups out there that are riding a similar train. Tatango comes immediately to mind, as little time has passed before once starting as they now only offer pay-to-play models. Personally, I think bad. Tatango is group texting somewhat “on steroids” – think Mozes.com but with much better hierarchal/admin/mod controls. Tatango’s biggest problem is its reliance on many other companies’ costs in providing SMS to third parties (iow, Verizon offering SMS “time” or blocks of message system use to Tatango as the 3rd party). Tatango’s two recent price increases prove that text messaging pricing is way out of hand both for individual consumer use and for startups, clever startups like Tatango. Tatango – although benefiting as does any other startup these days from any kind of angel funding – doesn’t need so much as money thrown at it, as better negotiated deals for SMS usage. I would bet money – that we don’t have – Jason Calacanis could figure out just what the heck can help a startup like Tatango and to help prevent what happened to Ma.gnolia.
And one more thing: In reference to backups, I agree with an earlier poster that ALL cloud services should offer their own backup system/engine so that their customers can have a local or secondary remote cloud copy – even on their own server or ISP leased server, for instance. You don’t have to be running MySQL servers just to store a copy of a MySQL database!
I would set the microphone to Omni (not figure-8) push it a little further away from you, still keep it between you and your guest, facing exactly between your two seated spots. It sounded like the mic was switched to fig-8 and was turned sideways to each of you – the pickup patterns were parallel to your shared plane, which made it sound lik only the camera mic was being used. Either that or it was only your camera mic and the table mic is just a prop!?
On the other hand, although the perspective is off (and I personally love the angle of the camera) the mic may have actually been to far away from you – that mic, and I am guessing here, is probably better served it is no more than 1 or 1.5 feet from you and your guest. 3 to 5 feet away is too far away for that type mic. Heil P40 perhaps?
I don’t blame you for what I lost, it was my information, and if it was important I should have had a backup. I loved how Ma.Gnolia was set-up, and nothing compares to it. Thank you for giving me that, and I do wish there had been a better backup systems, and we can all learn from this. Thank you for sharing. I tell everyone, computers are computers, and sometimes sh*t happens, and nobody ever understands why.
http://www.heelsunion.com
What a couple of twats!!!
I really appreciate u guyz …U accepted the failure in public …Few people directly shutdown the sites without any explanation which is frustrating… Ma.golia is my fav and i love to use its regularly …I missed out
Sri
Thanks, Larry and Chris for your great episode..pretty awesome
that’s why you invest in backups your twits…
hmmm….
get WD server drives one tera is almost 140$ cheap scapes…….
Whoever said that MySQL isn’t a good database should be ignored. OTOH, if any database doesn’t do what you need it to, then find one that does. I don’t care if it is SQLite, text files, gdbm, Oracle or DB2 running on MVS.
Hire a systems/enterprise architect if you need help with this stuff. There are many cost effective ways to be certain these things “just work” without too much human intervention. A 2 hour consult is pretty cheap compared to a failed business.
Lesson Learned:
1. Spend 5 minutes to do some research on how real IT systems work.
2. Use: LAMP (Linux, Apache, MySQL, PHP)
3. Dump: XServe, worldpress, and other lame products.
4. It takes more then a shiny web template to survive on the web.
5. Advertising makes money.. omg what a concept.
I was a great fan of Magnolia. Many fans like me are waiting for new arrival and new version of Magnolia. Honest people always get support. We are with you Chirs and Larry.
4 Trackbacks
[...] given Ma.gnolia’s recent catastrophe, we decided that episode 11 would dedicated to exactly what went down and why, and what lessons Larry has learned that others [...]
[...] Ma.gnolia has given up, and lessons learnt. I had started using it about a couple of years back, so I did lose some links, but most of them [...]
[...] homepage to break the news. This came a few days after he and Chris Messina used the forum of their Citizen Garden podcast to share more details about the technical aspects behind the breakdown, and the prospects for [...]
[...] applicable is the painful story of how Ma.gnolia went down because the only backup that was being done was a datab…. Ouch. Learn from others’ [...]