February 13, 2008
Listen up. Yes, you. My future somewhat disgruntled-or-maybe-just-frustrated co-worker or subordinate. (Subordinate? Did I just use that word? WTF?)
Did I just tell you to “reign in the schedule” and reiterate how important it is to the Powers That Be that we hit some arbitrary deadline?
THEN CALL ME ON IT!
As developers, we’re all tempted to give optimistic schedules. We like to believe in our own powers. And sometimes “two weeks” just sounds like a really, really long time for a task.
And the less real information we have about how long similar tasks took, the more we second-guess our estimates.
Then there’s the fact we like to sound agreeable in front of our bosses or business people.
And the more people–good, honest people, who themselves are feeling pressured–tell you how great it would be to get done earlier, the more we’re likely to cave.
And if we’re really inexperienced, or uncomfortable, or insecure? We stand no chance.
So what happens when a team has a culture of VIPs telling everybody how important it is to “reign in the schedule” every chance they get? Well, a couple of things.
First, people lie to your face. “Yeah, three and a half days. Yup. And by the way, can I just tell you how frigging awesome your voice is, sir? You should try out for American Idol.”
Second, people work 18 hour days and weekends.
The ones who value some semblance of being accurate estimators will hide this. They won’t tell you how much they’ve worked, and they’ll lie to themselves about it too.
These are the ones who burn out quickly.
Others will gloat about how much they’ve worked. And, yes, you give them some extra time off, or a token 4% raise, or maybe even a promotion, and they might not even mind it so much. Consciously.
But still, they’ll burn out. And, in the meantime, they’ll be so busy scrambling around working 3 jobs for the price of one that they won’t have a chance to improve processes, improve estimates, measure code quality, and all the other great things that pay off in the long term.
Oh, and the ones who stick to their guns because they’re experienced enough and confident enough and willing to walk out the door to uphold their professional reputation? Well, they become alienated. And they will walk.
Here’s what I suggest for any team leader, manager, or Big Wig presented with an estimate that doesn’t mesh with what we euphemistically call “business expectations” (the more accurate phrase is something I don’t think I can print).
Ask intelligent questions.
- Was a date set by business requirement before any work estimates were made? If yes, bzzt, the schedule can’t be trusted.
- Were the initial estimates and dates set by the developers who will do the work, or by an architect or manager? If the latter, something’s fishy.
- How many times did somebody say “we need to reign in this schedule”? Add one day to the schedule for each time somebody–anybody–said that. In some cases, yes, this means tripling the schedule.
- Did the estimates increase or decrease as the schedule was revisted and revised? Any schedule that doesn’t grow as developers break down the tasks, mentally, is suspect, unless requirements are eliminated.
- Does the schedule take into account average number of meetings in a week per developer? Better question: ask each person on the team how many hours they spend per week in meeting. For each person who doesn’t know, add 20% to the schedule.
- Does the schedule take into account fire drills and other emergencies?
- Does the schedule take into account context switches, down time, warm up time, etc.?
- Are the estimates based on previous schedules involving similar work by the same developers? If nobody’s been keeping track, this project should be the pilot. Which means no expectations. Oh, and increase the schedule by 30%.
- Are the efficiency constraints based on previous schedules? In other words, do we even know how much actual work people get done in a week? (Answer: it’s rarely 40 hours.) If not, add 25% to your schedule.
- Does the schedule include working long days, weekends, or holidays? If yes, bzzt, in the immortal words of Donny the Dingdong, YOU’RE FIRED!
Woops. So your “business expectation” was next Tuesday, and the original schedule said a week and a half after that…and now that we’ve delved into it, we’re looking at four months!
That’s what you get for developing such a nasty, unprofessional culture. Live with it.
OK, I’m feeling a bit snarky today. But everything I wrote is true.
December 26, 2006
While I’ve been on vacation, others have pointed out an SDForum presentation given by two senior eBay technologists, Randy Shoup and Dan Pritchett, on The Ebay Architecture: Striking a balance between site stability, feature velocity, performance and cost.
They tell a scalability story that is becoming more common, one which many people still find counter-intuitive:
- partitioned database schemas, based on usage patterns; mostly partitioned by primary key, with mapping tables where necessary;
- running with database slaves with various lag times;
- no stored procedures;
- no joins, sorts, or referential integrity constraints performed at the database layer: all this is done at the application layer, where CPU availability is easier to scale;
- no client-side or distributed transactions;
- homebrew frameworks as necessary (in eBay’s case, they use their own Java ORM framework, and their own connection pooling)
Aside from the tale of what they’re doing now, they provide an excellent history of eBay’s scaling problems through the years. It’s a great overview of what leads a company to these solutions.
If rules like “no joins at the database level” are becoming so common (or, as Joe Gregorio commented, it’s almost as if everybody is slowly implementing their own version of BigTable), why is it still counter-intuitive? I blame it on University Education. The approach to teaching databases at most universities is a lot like teaching multiplication tables to first graders: a lot of rule learning and regurgitation.
(There’s a very predictable 10-to-15 month cycle for a new hire at Webshots, which starts with blaming MySQL for all the world’s problems, moves through secretly inserting joins into production code, followed by resentment of the Establishment Authority, finally leading to enlightenment. Not that Webshots is anywhere near as good as the big guys when it comes to scaling.)
If you find this interesting, be sure to check out Scoble’s interview of Eric Billingsley, head of eBay Research, which I blogged about in October. Eric focused more on scaling search, and also goes into some of the history.
What I still find most fascinating about eBay is their approach to rolling out features: there’s a new release that contains 100K+ LOC (how much is auto-generated?) every two weeks, and they complete 300 new features per quarter. From what I hear from those inside eBay, this requires careful release management and putting in hours during night shifts, but it’s still awe-inspiring.
Finally, check out the summary and commentary by Frank Sommers over at Artima, which concludes with the following insight:
[T]he main message of the eBay presentation […] [is] the fact that eBay was able to meet the challenges of its growth with subsequent refinements to its system, all the while keeping the site operational.
The reason that’s interesting is because it suggests that you can start with almost any architecture—even with Perl or Rails or JSP pages—as long as you know how to migrate to the next step, and have the capability to do so, if and when you need to scale your app. That, in turn, suggests that the key test of scalability is not so much how each architecture stage scales, but how readily a company or an organization can move an application from one architecture step to the next. That indicates that scaling is as much an individual or organizational question as a technical one.
November 29, 2006
He reminds us that the flip side to innovating to get ahead (and stay ahead) is learning from, and shutting down, failures, but…even companies thick with innovative products are averse to following through:
If something does not work, the company needs to move on quickly. Failures need to be acknowledged, all possible learning extracted, and then the product should be eliminated.
This is not what happens. Instead, unsuccessful products are left up on the site to rot. Failed experiments become useless distractions, confusing customers who are trying to dig through the options to find what they need and frustrating any customer foolish enough to try them with the obvious lack of support.
He gives good examples from Google, Amazon, and Yahoo — big names in (web) innovation who, nevertheless, allow failed products and services to litter their navigation and soak up internal resources.
Even unsupported products still must be maintained operationally, security patches deployed and, to the extent they use shared libraries and APIs, evolved as the libraries and APIs evolve. Not to mention the customer support complaints from those sad individuals who stumple upon the unsupported product.
November 22, 2006
Flickr recently launched Camera Finder, a “joint” effort with Yahoo! Shopping. Another sign that Flickr is being (slowly) integrated into the Yahoo! Family. Good coverage by Mashable and Paul Kedrosky.
I like what they’ve done. Their camera finder is everything that Webshots’ Photo Gear Guide should have been (two years ago), but wasn’t…and won’t be.
A little history: Webshots’ Photo Gear Guide (or “Tech Guide”) was the first effort at integrating into CNET, a mere months after being acquired. It had a sordid history, and it could have been much better than it is. It’s now been mostly abandoned (which explains some of the empty content for editors who’ve left CNET).
Anyhow, some observations:
First, Webshots’ Photo Gear Guide is butt ugly (and I don’t mean the butt of Grace Park). It’s a nauseating mix of yellow and grey and red. Even two years ago before our new header, it was still pretty sickening to look at. It tried to have the look of a CNET property, with the cobranding of a Webshots property. Which is what half the pages really are (more on that in a bit).
By contrast, Flickr’s Camera Finder pages look exactly like any other Flickr page. The same prominent colors (even in the graphs!). The same kind of navigation.
Second, there is no original content on Webshots’ Photo Gear Guide. All the editors stuff and specs come from CNET Reviews, which is fine. But most of the links send you off to cobranded pages that aren’t even hosted by Webshots (and are even uglier–and now still have the old Webshots header). There is absolutely nothing to tie that content into the Webshots community. Nothing.
Flickr’s Camera Finder, on the other hand, has graphs of the popularity of cameras over time within the Flickr community. When you drill down into the cameras, you see photo search results of Flickr photos taken with each camera, sortable in several dimensions. It feels like another way to browse Flickr photos, and also a way to compare cameras.
Finally, Camera Finder makes you feel like you’re actually learning something. With PGG, you get editor reviews and specs and “best buys” based on price vs performance tradeoffs. All well and good, and I have no doubt that CNET’s reviewers do a better job than Yahoo’s Shopping editors.
But–because you’re not engaged within the Webshots community, you don’t really know how accurate those reviews are, or which camera is probably right for you. By utilizing Flickr’s community, you get a much better sense of not only the quality of photos produced by a camera, but also what kind of photographers are using which models. Find people like yourself. See what they’re using.
Of course, Webshots did not detect cameras two years ago (we do now, and it’s on every photo page unless the owner chooses not to display that information). I seem to recall that Nick and Narendra’s vision for the photo gear guide was exactly what Flickr produced. If you answer the question, “Why did it never quite materialize?” you’ll probably find the core of what’s been missing at Webshots the last several years.
With that said, I’m not sure Flickr’s Camera Finder will see significantly more page views than Webshots’ Photo Gear Guide. It’s very much a niche audience, that likely sees surges during the holiday season and maybe again in late spring. One way to counter this is to meaningfully link to the data from photo and member pages.
It will probably generate more revenue for Yahoo!, since they’re linking directly to Yahoo! Shopping and Yahoo! will get a cut of any sales (unlike CNET). Which partly explains why they put more effort into getting it right.
Anyway, a thumbs-up from me. The more you can leverage the interaction and personal choices made by your members, the better your community will be.
October 12, 2006
Early on, Eric talks about the design of their search engine in 2002. They had been trying to make a legacy in-house search work, and brought in all the major search vendors (Verity, Google, Excite, …) but found their search systems were no better.
Why is that? IR systems usually are built for query-time performance, not for real-time indexing. There are some very good reasons for that.
eBay was among the first to require not only near-real-time indexing, but also to provide large-scale faceted search. Listening to Eric, solving the near-real-time indexing problem using message queues and what-not was apparently easier than solving the faceted browsing problems: showing exactly how many results there are in any given category, where the categories change with every query. (See also my previous post on faceted search and indexing effiency for a recent cool attempt to solve this.)
Like many things in this changing world, what was once large-scale (20M items) and high-performance in 2002 has quickly become expected behavior, not only at eBay but at most social networking sites as well. For example, tag browsing is expected to be near-real-time. Blog search is expected to be near-real-time: Technorati indexes more blog entries every week than eBay had in total in 2002.
Later, Eric demos a couple of ideas in the oven, including what appears to be dynamic ranking of results (reordering of results based on click streams).
One thing that sticks out is he talks about eBay having a full site release every two weeks, and he describes this as “massively high frequency.” In the Web world, I think that is an exaggeration. Weekly–or even daily–releases are more and more the norm. Java shops have a more difficult time keeping up than Perl, Python, PHP or Ruby shops (though the Java shops tend to be larger).
What is probably unique is that he says they have a highly regimented release schedule, which, presumably, means no slips and no code freezes. That’s hard to do with a company the size of eBay (in terms of number of developers).
The interviews goes on about operational issues and a nice-looking (but probably not very useful) social interaction browser.
Interesting fact from near the end: eBay has 768 back-end servers just serving search queries. Documents (items) are split into one of 16 buckets, each of which is served by a cluster of 16 servers, and there are 3 full redundant versions of the whole system, each capable of taking all the traffic in a crunch.
Worth a watch.
October 1, 2006
If you haven’t read it already, Googler Steve Yegge rants about Agile development and paints a rosy picture of life at Google. Go ahead, read it. (My second favorite rant of the year, but more informative than my favorite.)
Commenters have already critiqued his post. To summarize:
- Google’s products are rarely profitable; Google is acting much as a VC firm, except their bets are almost all in-house
- Google hires from the rightmost end of the Bell curve, which, while great and representing the kind of people most of us would want to work with, isn’t universally applicable (by definition)
- Google has yet to weather a downturn; it is young and its methods are unproven; they’ll likely modify their practices during lean times, as all companies who survived the late 90’s did
With that out of the way, Steve makes some great points and has helped me pinpoint some of my own frustrations. I think, ultimately, what Google has found works for them addresses the same frustrations.
February 11, 2006
Dave Winer points to DFL, which highlights the last-place finishers in Olympic events. Apparently it was mildly popular during the Athens games. My only wish is he could conduct interviews with the last-place finishers, and provide some sense of what kind of chance they had in the first place.
This appeals to me in the same way that my former pseudo-hobby of collecting memorabilia for sports stars who never were–or who were, then weren’t–appeals to me. There’s a story behind every star who didn’t excel. No matter how promsing you are, you still have to close the deal with a lot of hard work. Some take to drugs. Others let the fame get to them. Still others get injured, or have family problems. Some even find that the game changes beneath their feet. And still others–well, they end up simply outclassed.
Usually, of course, even the ones who never fulfilled their promise are still better than 99% of the detractors who wish they’d had the same chance.