Briefly following up on the Twitter situation from my last post, Michael Arrington of TechCrunch posts a short list of extremely pointed questions for Twitter.  In it he says...

So I have a couple of questions, too, based on a couple of discussions I’ve had with people who say they’ve seen Twitter’s architecture.

• Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
• Do you really have a grand total of three physical database machines that are POWERING ALL OF TWITTER?
• Is it true that the only way you can keep Twitter alive is to have somebody sit there and watch it constantly, and then manually switch databases over and re-build when one of the slaves fail?
• Is that why most of your major outages can be traced to periods of time when former Chief Architect/server watcher Blaine Cook was there to sit and monitor the system?
• Given the record-beating outages Twitter saw in May after Cook was dismissed, is anyone there capable of keeping Twitter live?
• How long will it be until you are able to undo the damage Cook has caused to Twitter and the community?

Now, to a certain extent this is Mr Arrington realizing what type of posts sell and taking advantage of that fact at a normally slow time (the weekend).  But there are a couple of interesting issues here.

On the technical issue: If half of this list is true than Twitter's infrastructure was built by a complete amateur (and that's being generous).  Or maybe more accurately a programmer who didn't understand the first thing about IT infrastructure.  Whatever the case it shows a business with no idea where to turn when it came to building a solid infrastructure.

To my eyes this is endemic of an industry that has (a) allowed academia to abandon relevant experience and (b) refused to regulate itself with certifications. 

Computer Science programs focus almost completely on theory while Information Systems degrees focus largely on business so both create candidates with huge gaps in their knowledge.  Beyond that the only certifications that exist are from individual vendors and those are more marketing than they are verification of knowledge.  So you get an industry where it's impossible to identify who knows what they are doing (unless you ARE someone who knows what they are doing)

Everyone from Doctors to Plumbers have professional certifications to verify their knowledge.  That's why you don't need to be a Doctor to hire one.  The fact that Twitter, one of the most scrutinized companies right now, has such a painfully insufficient setup suggests it might be time for the IT industry to look into ways to verify its members.

On the Blaine Cook issue: Since the first serious tech job I ever had there has been a constant in the industry.  Companies always insisted on Non-Disclosure agreements and tech workers always hated them.  Recently that policy has started to lift and Mr. Cook is just some of the fall out. 

After seeing the damage he's done I can't help but wonder if Twitter isn't a lesson in why they are necessary.  At least after termination (you could have someone agree to it at the time they are hired).  Transparency is great until your ex-employees turn on you and once that happens they can do some serious damage to the company as a whole. 

NDAs after termination keep your ex-employees from destroying the morale of your current ones. 

Anyway, its about time for me to get ready to go out for the night but I wanted to throw out those two quick impressions before they got forgotten amidst other weekend activities. 

So this is kind of funny...

On February 8th, 2008, Israel of AssetBar.com wrote a really informative post entitled "Twitter Proxy: Any Interest?"  In it he pointed out this fundamental problem with Twitter...

Nothing is as easy as it looks. When Robert Scoble writes a simple “I’m hanging out with…” message, Twitter has about two choices of how they can dispatch that message:

1. PUSH the message to the queue’s of each of his 6,864 followers, or
2. Wait for the 6,864 followers to log in, then PULL the message.

 

The trouble with #2 is that people like Robert also follow 6,800 people. And it’s unacceptable for him to login and then have to wait for the system to open records on 6,800 people (across multiple db shards), then sort the records by date and finally render the data. Users would be hating on the HUGE latency.

This post was later quoted by Dare Obasanjo which led to an angry response from Robert Scoble where he said...

First of all, Twitter doesn’t store my Tweets 25,000 times. It stores them once and then it remixes them. This is like saying that Exchange stores each email once for each user. That’s totally not true and shows a lack of understanding how these things work internally.

Second of all, why can FriendFeed keep up with the ever increasing load? I have 10,945 friends on FriendFeed (all added in the past three months, which is MUCH faster growth than Twitter had) and it’s staying up just fine.

Now, with no offense to Scoble, this is a fundamentally ignorant assessment of what is actually going on.  A fact that almost every technical minded person pointed out to him.  The best of those posts, imho, was from Nick Halstead where he says...

In a recent post Robert Scoble tries to explaining how twitter works by saying that twitter is using some form of ‘pivot table‘ - (my terminology for what he explains) and says that a model that others have put forward (i.e. a de-normalized system of inserting messages into everyones queues) was akin to microsoft exchange, now these two examples are so horribly not connected - and I won’t rant about how BAD exchange is efficiency wise, but please Robert do not get into any technical arguments please.

(Its a great post that the above quote doesn't do justice so everyone should check it out)

This led to the same question being presented to the authors of the official Twitter technology blog.  Their reply was to basically confirm what everyone had been saying...

charles asks if there's anything users can do to lighten our load. The events that hit our system the hardest are generally when "popular" users - that is, users with large numbers of followers and people they're following - perform a number of actions in rapid succession. This usually results in a number of big queries that pile up in our database(s). Not running scripts to follow thousands of users at a time would be a help, but that's behavior we have to limit on our side.

A response that Scoble quickly took offense to saying in his link blog...

“This is total bullshit. Why do I have 11,556 subscribers on FriendFeed, I'm FAR FAR FAR FAR more active on FriendFeed, and yet FriendFeed never has gone down on me? Also, Twitter went down at its first SXSW before I had a ton of followers there. Twitter has major problems, they still don't have a good engineering answer, and so they are blaming their most popular users. Great. We get the message. We'll go someplace where there's a good engineering team. You know, the guys who invented Gmail and Google Maps? They are the ones behind FriendFeed. See ya Twitter!”

So there you have it, a little background on what has become a huge mess for Twitter. 

Honestly, I don't have much of an opinion on the human drama part of this (other than to find it pretty amusing).  Is Scoble being unreasonable?  Absolutely.  But that's somewhat understandable when you consider the fact that everyone is pointing their fingers at him when he essentially did nothing wrong.  The truth is, Twitter loved Scoble signing up 25,000+ followers when it meant drawing users in to the service. 

So for him to be slapped on the hand now for it is a bit obnoxious. 

On FriendFeed, there are two issues there.  One,  the model by which people monitor other people's FriendFeeds is not like the model that Twitter uses which lowers their back end problems and two, FriendFeed has far fewer users.  If they were trying to deal with as many users as Twitter I suspect they'd be having problems as well. 

One final point I'd like to make is regarding the root cause of Twitter's woes.  A lot of people have tried to lay this on the feet of Ruby on Rails which is a bit unfair.  With that being said, Twitter's problems do belong at the feet of the "Rails Philosophy" which was set down by its creators 37Signals.  Here's a quote from their e-book "Getting Real"...

You don't have a scaling problem yet

"Will my app scale when millions of people start using it?"

Ya know what? Wait until that actually happens. If you've got a huge number of people overloading your system then huzzah! That's one swell problem to have. The truth is the overwhelming majority of web apps are never going to reach that stage. And even if you do start to get overloaded it's usually not an allor-nothing issue. You'll have time to adjust and respond to the problem. Plus, you'll have more real-world data and benchmarks after you launch which you can use to figure out the areas that need to be addressed.

I've never thought much of the 37Signals gang and it is quotes like the above one that are the reason why.  Putting off your most difficult technical tasks until later is an utterly stupid thing to do and is, as Twitter is now finding out, disastrous if you can't quickly address the problems.  37Signals wouldn't necessarily know that though because their most popular application has fewer users than a program I wrote at 19.  Yet they still speak as and are treated like they are leading experts in application design. 

They're not. 

Not only that, a lot of their advice is outright bad as this Twitter solution proves.  This is all symptomatic of a larger problem on the web which is bloggers' not questioning who they treat as an authority.  Anyone who realistically looks at 37Signals will see that they are still a fairly small development firm.  That doesn't mean they haven't done some impressive things or that their opinion has no merit at all.  But it needs to be put in context and people don't seem to be doing that. 

Had the Twitter architects done that they wouldn't be in the situation they're in now. 

Addendum:  This occurred to me as I was hitting the publish button of the above post.  When you go to the bank for a new business loan what is the first thing they ask you for?  A 2-Year Plan.  That's because no sane institution lends money to anyone who has no idea where they are going or how they plan to get there.  Given that I ask you, doesn't the same apply to technology issues and in particular scaling? 

A few of e-mails over the long weekend asked if I had an opinion on the new OLPC announcement (being it seems to be right up my alley given my job).  The truth is, I had been trying to avoid the topic because I find the difference between the potential of the idea and the reality almost painful.

For those unfamiliar, OLPC stands for One Laptop Per Child and its an organization that created a cheap laptop for kids in developing nations (though most times OLPC is used to refer to the laptop itself which is officially named the XO-1).  It has wireless and flash memory but no hard drive and runs on a Linux operating system.  The organization recently announced a new version due out in 2010 that would use a touch screen (see picture below)

Look, as much as I hate to say it, the OLPC has never been a good idea.  In fact, my entire career is an example of how its not a good idea. 

You see, I do what I do because there's no money in helping kids.  Simple as that.  No one spends money on helping kids because no VC will fund a startup that helps kids because most of the kids who need help don't have the money to pay for it.  Which means almost no good software gets written to help kids. 

You ever wonder why educational software is so lousy? 

Its because it was done on a shoestring budget by a programmer who was probably working for cheap.  Again, there's simply no money in helping kids. 

Which brings us back to the OLPC.

OLPC uses a rare version of Linux with a custom environment called Sugar on top of it.  This means that software has to be specifically written for the OLPC and that just isn't going to happen.  That developer I mentioned before, the one who built cheaply made children's programs, isn't going to learn Python, Sugar and Linux to write for a laptop that only poor kids are going to have.

So while I love the idea of every kid in the world carrying around their own mini-notebook I realize that no software is going to be written for it unless its bootstrapped onto something else.  Giving kids a PC with no software is about as bad as giving them no PC at all. 

That, by the way, is where products like Asus' Eee PC come in.  The Eee PC is a $399 "mini-pc" running a full version of XP Home.  It isn't rugged-ized which is a problem when dealing with kids and it costs $199 more but it can run any Windows software and comes with a few niceties like a built in webcam. 

Plus, the OLPC organization has never been great with price.  The OLPC 1.0 was supposed to come in under $99 but it ended up costing $200.  Now they say they have a more advanced touch screen model that's going to cost $78 and we're supposed to believe them?  Not likely...

Anyway, long story short, the OLPC is a nice thought but it will never get traction.  I can't wait for the day when every kid can have their own notebook but that isn't going to happen through the OLPC. 

There was a book published in the late 90s entitled "Barbarians Led By Bill Gates" which was a fairly bitter tale of how backwards one employee (co-author Marlin Eller) felt Microsoft was in the early days.  Unfortunately, the book suffers from all the negativity. 

But one chapter really shines and that is the Chapter on Pen Windows.  Pen Windows was a project attempting to add a Pen interface to Windows 3.1 and it was a project that Mr. Eller worked on directly.  At the time a company named GO was making a lot of noise in tech circles by convincing people that Pen based interfaces were the next big thing.  So Pen Windows was Microsoft's response to that. 

I should mention that, by "Pen interface", we're not talking about drawing on the screen like a tablet but instead about drawing on a Desk pad as a mouse replacement. 

In the end, Pen Windows was a failure and GO went completely out of business.  As a final assessment of the situation Mr. Eller quotes himself as saying this to his partner on the project...

Greg, Look.  This wasn't a thing about making money.  This was all about 'Block that Kick.'  We were on the special team.  We were preventing GO from running away with the market.  That was our job.

Look, your background is in applications, you have to ship the application.  My job is in systems.  Systems, for much longer on, has been completely 'Don't let anybody else steal DOS from us.'  That's all we're doing.  We weren't trying to sell software, we were trying to prevent other people from selling software. 

From my view, Pen Windows was a winner.  We shut down GO.  They spent $75 million pumping up this market, we spent $4 million shooting them down.  They're toast!

I bring this up because news broke yesterday that Facebook will be Open Sourcing their platform in the next week and I think the above quote does a lot to illuminate why. 

The best way for Facebook to eat away at their competition is to keep their competition from innovating.  The easiest way to do that is to hand out the Facebook platform.  That way your competition thinks they're getting the holy grail when in fact they are stunting their own growth.

If a Social Network works exactly like Facebook than there is absolutely no reason for users, developers, et al to leave Facebook.   So while Facebook's competitors probably see the ability to integrate the Facebook platform into their service as a huge opening its actually just Facebook "preventing other people from selling software" 

Plus Facebook gets to kill off Google's OpenSocial in the process.  Not bad for a day's work. 

I think its important to remember that the old rules still apply in the software business.  People might pay for software with their eyes rather than their wallets but they are still paying for software and companies still need to compete for that currency.  I might not like tactics like this one but they show Facebook is paying attention to the past and in my book that puts them ahead of the competition. 

Its a really good idea to get away from the Internet every once in a while.  Especially after spending a day immersed in it. 

There's no better cure for blog-itis than spending some time in a world where a conversation is more than two people yelling at each other from across Techmeme and a friend is more than someone you accept to boost your "followers" count. 

But at the same time it is good to be back.  Life is good, the Pizza is plentiful, and I never truly realized how soft my "firm" mattress actually is.  Anyway, back to normal posts (at a normal pace) tomorrow. 

I'm leaving for the great outdoors in an hour or so but I wanted to put a period on the whole Ariel Waldman thing.  It started out as just a post but more and more the people I see on the other end of things (just about everyone on Techmeme) scare me.

Not because they disagree with me but because I realize the opinions they espouse don't match up with the values they claim to have.  You endlessly hear blogger's go on and on about the evils of censorship but the second someone says or does something they don't like its open season on that person. 

Who needs values if you have unbridled emotion I guess...

I keep hearing the argument of...

"What if it was your Mother/Daughter/Wife/Sister?  Wouldn't you want it taken down then?"

and I have to wonder, do these people know what having values even means?  If your values change when applied to those you care about than you simply have no values at all. 

Moreover if you are willing to make exceptions to your values every time someone does something you don't like you have no values at all. 

I was once told that most Germans living in Nazi Germany were people who put a high premium on morality.  The problem is they were willing to make exceptions when it came to people they didn't like...and they didn't like the Jews. 

Values are based on judgment, dislike is based on emotion.  When you allow emotion to overpower your judgement you open yourself up to committing the worst evils that the world can produce and, quite frankly, shutting up a name caller isn't worth that. 

I'm a little annoyed... 

You see, a couple years back Microsoft announced that they, like Google, would start indexing books and professional journals which were only available in print and then integrate those results into a specialized search engine.  I'm a big supporter of this movement because I believe its very important to index those printed documents so they don't get lost in the new digital age. 

Back to Microsoft, To do this they joined with the Open Content Alliance, a non-profit group dedicating to archiving this sort of thing.  At the time it was a coup because Google had chosen to go it alone with their book search making Microsoft look like the good guy. 

Good PR all around, Until Today...

This also means that we are winding down our digitization initiatives, including our library scanning and our in-copyright book programs. We recognize that this decision comes as disappointing news to our partners, the publishing and academic communities, and Live Search users.

Given the evolution of the Web and our strategy, we believe the next generation of search is about the development of an underlying, sustainable business model for the search engine, consumer, and content partner. For example, this past Wednesday we announced our strategy to focus on verticals with high commercial intent, such as travel, and offer users cash back on their purchases from our advertisers.

I have to say it...what a bunch of dopes.

Look, I'm for a sustainable business model as much as the next guy but when you have nearly 80,000 employees and make $51 BILLION per year it pays to look at the big picture.  The big picture where the cost of a few $35,000 a year employees is worth 10 times that amount in good will. 

I mean, this doesn't require highly paid programmers.  The technology behind the book scanning is long since worked out.  Its simply an issue of caring enough to keep some low level clerical employees on the job.  I'd think Microsoft would do that just to keep their search index growing. 

As Danny Sullivan of Search Engine Land put it...

Gosh, Google somehow seems to be able to run a sustainable business model and devote some energy and resources into indexing books and scholarly information, even if those generate little to no revenue. They do it in part because they think it's good business to provide all types of searches, not just those that will earn them money.

In the middle of a search war, I can understand that a "distraction" like book and academic search might seem like something to Microsoft that has to go. However, Microsoft's not hurting for cash to keep it up, if it wanted.

I mean what could Microsoft Execs possibly be thinking here?  I've been to Microsoft, I've seen parts of their org. chart, you can not tell me that cutting book search belonged at the very top of the cost cutting list.  This program could be funded for the cost of a couple useless Program Managers and Microsoft would probably be better off for having fired the Program Managers

(not that all Program Managers are useless but there are plenty working there who are)

Honestly, what a bunch of idiots.  If anyone with any power at Microsoft is reading this, follow my advice: Fire whoever made this decision and use their salary to keep the program going.  THAT would be a good business move. 

I've been on the Internet since around 1st grade and in that time I've been a part of a lot of communities.  What I've noticed is that building a  successful community often means allowing elements into it that you don't like or approve of.  The extent of that depends on how open a community you want but as a general rule the more you want people to freely express themselves the more you have to allow unsavory elements in.

Keep that in mind when reading this excerpt from a post by Ariel Waldman 

In June 2007, I unfortunately found myself on the receiving end of multiple accounts of harassment from a user on Twitter. When the user started using my full name in their harassing tweets, I reported the harassment as a form of cyberbullying to Twitter’s community manager and received a response that let me know they cared about the situation

The harassment continued throughout the course of 2007. Since Twitter and I had an open dialog started, I would periodically report cases of continuing harassment (some of which spread between Flickr and Twitter). Twitter would take no action while Flickr would immediately ban and remove all traces of the harassment.

The story goes on but the gist is that Twitter refused to ban the aforementioned name calling user despite Ms. Waldman's continued pleas for relief. 

I think my first reaction to this was the same as most people's.  Ms. Waldman seems like a nice person, nice people don't deserve to be called names, anyone who calls a nice person names is a bad person, bad people deserve to be banned. 

What's there to debate? 

Well, a lot as it turns out.  The question here isn't "is the person doing the name calling bad?"  I suspect everyone on the Twitter side dislikes this person and wishes they weren't using the service.  The question though is, short of things that are against the law, does Twitter want to become a service that imposes management's values on its users?

Values are not absolute and one person's harassment is another person's "open expression".  I completely understand where Ms. Waldman is coming from and wish she wasn't being subjected to this but again, an open society can be unpleasant. 

Honestly, I have to grudgingly admit I don't consider the quoted name calling harassment.  I don't like it, but since Twitter allows you to block users I think the name caller would have to rise above simple posting to constitute harassment.

If Ms. Waldman blocked the person and they created another account to continue the name calling you might have a case for harassment but as told the story doesn't seem to rise to the harassment level.  I mean, its the Internet, where a lot of people encourage others to hate them while outwardly decrying it to get publicity.

Values are relative

This comes back to what I was trying to say in my opening paragraph.  If you want your service to be an open platform that represents society than you have to permit the elements of society that you don't like.  That includes bad people who call nice people mean names.  I wish this name caller would just go away and leave her alone but I don't think I have the right to make them.  I suspect the people at Twitter feel the same. 

Addendum: For the record, I didn't mention Ms. Waldman's primary claim which is that Twitter should ban the person for violating their Terms of Service agreement which says users won't “abuse, harass, threaten, impersonate or intimidate other Twitter users."  Honestly, with no offense intended, I don't consider this argument to have much merit.  ToS agreements are to protect the company and they are free to enforce them as they please.  Most companies allow violations of their Terms of Service all the time and we all know it. 

Addendum 2: I didn’t realize it when I posted but it turns out the account Ms. Waldman wants shut down is in fact an account in which people can post anonymously.  So it isn’t a person with a twitter account it’s one of many people who use the account to post anonymous messages.  I’ll leave it up to you to decide whether that changes anything but I thought I should put it out there.

Sorry for the flurry of posts today, we were supposed to leave for a camping trip but those going with me had to stay an extra day leaving me nothing to do (I thought about working but can't bring myself to after the record 90 hour work week I just did to make up for all the time away this month)

Anyway, with nothing to do but sit here and waste the day away online I've found a little extra inspiration. 

(I also learned how to juggle though admittedly not well)

For anyone who hasn't heard, Microsoft has a new plan to gain search market share which is to share the wealth.  Basically they will pay you "cash back" every time you use Windows Live Search to find a product and then buy that product.  Here's a quote from Techcrunch that pretty much sums up the situation...

Google’s search dominance is growing, and everything Microsoft has historically thrown at them has done nothing to slow them down. This new approach is both desperate and brilliant. Desperate because Microsoft is giving away most of the search revenue to get market share gains. Brilliant because they have such a small share of search revenue today that they have little to lose, and they are hitting Google hard in their core business.

I also think this is brilliant but for a completely different reason. 

Though I'm not personally a fan of Live Search I know many people who seem to think its as good as Google.  It certainly isn't dramatically worse.  So the question really isn't a technical one at this point.  Google essentially has two things keeping them as the market leader right now. 

1.  Marketing: This is the obvious one, Google's name is synonymous with web search at this point. 

2.  Familiarity: This is the less obvious point.  My theory here is that even if Microsoft produced dramatically better results people would still think Google is better because they are so familiar with that experience.  Think of the people who still insist the sound quality on Vinyl is better than a CD and you'll start to see what I mean.  Except in this case, unlike Vinyl vs. the CD, Microsoft can never scientifically prove to people that their search is more effective.

These two things are a powerful one/two punch for Google.  They basically mean that people will always go to Google first and will always think Google results are better even if they aren't. 

What Microsoft needs right now is a way to lure Searchers to them and get them to use Microsoft search long enough to grow accustomed to it.  Paying people is as good a way as any to accomplish that.