February 27, 2008
Development practices are all about dependencies.
Usually, we think of things we depend on. “Crap, I forgot to upgrade to commons-logging 1.1.1.”
Sometimes, we think of others using our code. “I better make sure this unit test consistently passes, or that new guy I just made an offer to is going to hate me and post bad things about me online.”
But there are times when what are perfectly justifiable architectural decisions in most contexts are the wrong decision when other teams are depending on you.
Let’s say you’re a central credit card processing gateway for various teams throughout your organization. For perfectly valid reasons, you’re going simply perform an auth and return–and then try to capture later from a batch job. Maybe that batch job runs 5 seconds later, maybe 60, maybe it’s broken for 2 hours. Most of the time, it doesn’t really matter.
But here’s the gotcha: what if other teams want to run unit tests (OK, “automated integration tests using unit testing tools”)? What if there are actions that a user is forbidden from taking during this delay, which is usually OK since they’re not likely to take those actions anyway (e.g., cancelling 5 minutes after subscribing to a service).
Suddenly, your scalable architectural decision makes everybody relying on you think you’re a flake, because their unit tests often fail for no apparent reason.
Or else they have to persist the last 24 hours worth of test accounts, and keep track of them.
All the moreso when you take the common lax approach to maintaining your development and staging environments. Who really cares if the batch job doesn’t run in dev for a few days? Well, that guy over there trying to meet a deadline who just wasted six hours because he thought the problem was in his code…
You’re an architect who has a neat new idea for how to deploy configuration changes to production. It’s completely different from what everybody else is doing, but it’s theoretically perfect, and requires less code than any alternative.
Of course, often what’s simplest from a coding perspective can waste many hours of your operations’ guys valuable time. Maybe it’s hard to understand. Maybe it’s hard to figure out what’s going wrong if the server crashes. Maybe it’s so different from the other deployments they have to manage that they spend thirty minutes relearning every time they have to deal with it. Maybe it decouples what ought not be decoupled, and it’s near impossible to get version dependencies just right between different components.
The difference from the first case is, the operations and database engineering teams usually have enough political clout (and, um, willfulness) to make you sit down and talk to them first.
You’re the kind of cowboy programmer who (rightly) believes that there’s no such thing as too much information when something’s wrong. So you start plopping down error messages and stack traces left and right, for every conceivable error.
Six months later, your teammate gets called at 3AM that the web site is on its knees. So, he starts digging into the logs…only to discover tens of thousands of lines of stack traces flashing by, and no easy way of distinguishing real problems from red herrings.
Sure, you’re right, that’s a team problem–there really ought to be good tools for examing all those logs in real-time, and said tools should allow you to filter out common problems that aren’t occuring with greater than usual frequency. So it’s not your fault, right?
Wrong. Unless and until your team have such tools, your trusted team mates will only get ticked off at having to wade through the very muddy, very swift river.
So, who’s depending on you?
Not that I’ve been counting and cataloging all my mistakes (that would be too depressing), but most of my mistakes have been attributable to not thinking about everybody who depends on me. And many of my team mates’ mistakes are the same.
It’s a surprising thing, but consider this: a junior programmer with absolutely no experience in a fast-paced, scalable environment has two options. He can think of himself and do the most expedient thing, maybe use the most cutting-edge technology to grow his mad skills. Or he can think of the people depending on him, and, even in the absence of a good mentor or even specific advice, bring the project into a state that will help them when they need it most.
If he chooses the latter, 99% of his inexperience becomes mostly harmless.
And still, experienced people often choose the former. And 99% of our experience becomes mostly worthless, if only for a few hours a week.
February 22, 2008
Dietrich Kappe recently posted why he doesn’t hire architects. He argues that architects are a product of large companies.
I tend to agree. Even in organizations with the architect title/role, they should spend at least 50% of their time programming. And engineering managers, 20%.
There are lots of good reasons. Can you really afford to pay somebody a higher salary for producing less? Can somebody who’s not in the trenches really make wise decisions about how to navigate those same trenches? Are people who don’t have to live with the consequences of their decisions really going to make better decisions? Will your organization survive a technical leader whose incentives are often opposite those of the rest of the technical team?
Between the fire-fighting and bug queue management and meetings and long-term plannings and code reviews and test plan reviews and political navigation and coding for multiple projects, you can easily spend 50% of your time just context switching.
You really need somebody who can step back from the fray, and keep steering the team in the right direction. Your team presumably has values (like not working weekends, pushing back against unreasonable pressure to meet unrealistic schedules, gradually reducing the cyclomatic complexity of your code, and so on). If you’re in the thick of it, it’s hard to not compromise, or even forget, those values.
I consider this an organizational optimization problem. How do you distribute the work effectively, so that every person is just below capacity, given that the cost of context switches is relatively high and varies depending on the type of tasks being switched off?
February 17, 2008
Julia was in a bit of a pickle. She’d gotten herself and her children in far over their heads. Her job was paying just above minimum wage, her savings was nearly depleted, she was $100,000 in credit card debt, and what’s worse, her interest-only ARM payments had just ballooned the previous month and she would certainly lose her home soon.
Julia needed a roommate to help her with her payments.
She screened at least thirty people over the next few weeks. Some responded to her Craiglist ad, others were referrals from her co-workers.
Most of the applicants were too inexperienced. There was the handsome guy who knew a lot about money and was on his way to be an investor on Wall Street…but he was just out of college, and had never actually lived with anybody before. Julia needed somebody with more experience.
Finally, she narrowed the field down to three candidates.
First was Matt, the wealthy old guy who always dressed right and said old-fashioned things like “yes, ma’am” and “have a pleasant evening.” He admitted to a peculiar fetish for spying on young women when they slept, but he’d never actually harm anybody.
Second was Robert, the middle-aged divorcee who was also determined to turn his life around. And what’s more, he promised to pay her twice what she’d advertised, and all he asked in return was for exclusive use of the kitchen every evening between 4-10PM.
Third was Herbet. He’d been living with people for twenty years. He started off in petty theft, stealing pens and clothes hangers from his roommates. Later, he moved on to stealing underwear and taking hidden photos and posting them online. More recently, he’d been convicted four times of kidnapping each of his past four roommates. Julia couldn’t argue with his experience, and he promised that, within a year’s time, he’d be paying all of Julia’s bills for her–her mortgage payments, credit card payments, car payments, health insurance, everything. You see, he was good at playing the state lottery, and knew just when, statistically, he could expect a big payout.
Of course Julia chose the one with the most experience, Herbert.
She and her five children haven’t been heard from since. Likely, they’re out partying hard in the Bahamas thanks to her fine choice in roommates.
February 13, 2008
Listen up. Yes, you. My future somewhat disgruntled-or-maybe-just-frustrated co-worker or subordinate. (Subordinate? Did I just use that word? WTF?)
Did I just tell you to “reign in the schedule” and reiterate how important it is to the Powers That Be that we hit some arbitrary deadline?
THEN CALL ME ON IT!
As developers, we’re all tempted to give optimistic schedules. We like to believe in our own powers. And sometimes “two weeks” just sounds like a really, really long time for a task.
And the less real information we have about how long similar tasks took, the more we second-guess our estimates.
Then there’s the fact we like to sound agreeable in front of our bosses or business people.
And the more people–good, honest people, who themselves are feeling pressured–tell you how great it would be to get done earlier, the more we’re likely to cave.
And if we’re really inexperienced, or uncomfortable, or insecure? We stand no chance.
So what happens when a team has a culture of VIPs telling everybody how important it is to “reign in the schedule” every chance they get? Well, a couple of things.
First, people lie to your face. “Yeah, three and a half days. Yup. And by the way, can I just tell you how frigging awesome your voice is, sir? You should try out for American Idol.”
Second, people work 18 hour days and weekends.
The ones who value some semblance of being accurate estimators will hide this. They won’t tell you how much they’ve worked, and they’ll lie to themselves about it too.
These are the ones who burn out quickly.
Others will gloat about how much they’ve worked. And, yes, you give them some extra time off, or a token 4% raise, or maybe even a promotion, and they might not even mind it so much. Consciously.
But still, they’ll burn out. And, in the meantime, they’ll be so busy scrambling around working 3 jobs for the price of one that they won’t have a chance to improve processes, improve estimates, measure code quality, and all the other great things that pay off in the long term.
Oh, and the ones who stick to their guns because they’re experienced enough and confident enough and willing to walk out the door to uphold their professional reputation? Well, they become alienated. And they will walk.
Here’s what I suggest for any team leader, manager, or Big Wig presented with an estimate that doesn’t mesh with what we euphemistically call “business expectations” (the more accurate phrase is something I don’t think I can print).
Ask intelligent questions.
- Was a date set by business requirement before any work estimates were made? If yes, bzzt, the schedule can’t be trusted.
- Were the initial estimates and dates set by the developers who will do the work, or by an architect or manager? If the latter, something’s fishy.
- How many times did somebody say “we need to reign in this schedule”? Add one day to the schedule for each time somebody–anybody–said that. In some cases, yes, this means tripling the schedule.
- Did the estimates increase or decrease as the schedule was revisted and revised? Any schedule that doesn’t grow as developers break down the tasks, mentally, is suspect, unless requirements are eliminated.
- Does the schedule take into account average number of meetings in a week per developer? Better question: ask each person on the team how many hours they spend per week in meeting. For each person who doesn’t know, add 20% to the schedule.
- Does the schedule take into account fire drills and other emergencies?
- Does the schedule take into account context switches, down time, warm up time, etc.?
- Are the estimates based on previous schedules involving similar work by the same developers? If nobody’s been keeping track, this project should be the pilot. Which means no expectations. Oh, and increase the schedule by 30%.
- Are the efficiency constraints based on previous schedules? In other words, do we even know how much actual work people get done in a week? (Answer: it’s rarely 40 hours.) If not, add 25% to your schedule.
- Does the schedule include working long days, weekends, or holidays? If yes, bzzt, in the immortal words of Donny the Dingdong, YOU’RE FIRED!
Woops. So your “business expectation” was next Tuesday, and the original schedule said a week and a half after that…and now that we’ve delved into it, we’re looking at four months!
That’s what you get for developing such a nasty, unprofessional culture. Live with it.
OK, I’m feeling a bit snarky today. But everything I wrote is true.
February 3, 2008
This Tuesday, I will do two things I’ve never done before.
First, I will vote in a primary. (I’ve showed up at the poll during primaries, just never voted in a primary race.)
Second, I will vote for a Democratic Presidential candidate.
I am registered as unaffiliated and, in California, that means I can vote in the Democratic primary if I choose. This year, I will choose to do so in the hopes of doing my part to see to it that Obama, and not Clinton, gets the Democratic nomination.
And if he does, he will get my vote in November, too.
My Political Philosophy in Six Paragraphs
Not to digress too much, but, briefly, here’s where I’m coming from.
What defines a government is its political processes. Do the processes that exist provide some degree of epistemological certainty that its powers are being used legitimately? Are violations of the processes few, brought to light, and corrected for?
That’s not to say that any of its policies are right, necessarily, just that the decisions on how to use that power are sound.
More practically, there are three real problems with the concentration of power. These effect private organizations too, but governments almost always have more power concentrated in them than any private organization.
The first is corruption. Power attracts corruption, and can then be used to increase corruption. People who want power will do anything to get it. So you need a government that exposes corruption and corrects for it. You’re never going to eliminate it, and that’s OK–as long as sound processes are usually followed in using power, some amount of corruption can be tolerated and worked around.
The second is consequence. Any action that a powerful government takes has far-reaching consequences. So you need a deliberation process that takes this into account, and you need to implement your policies in a way that makes all consequences–intended and unintended–as transparent as possible. Then you need to alleviate the negative consequences.
The third is momentum. Once you’ve taken an action, it’s mighty difficult to stop reinforcing that action, and near impossible to take a completely different course even if the consequences are disastrous. And while the market can react quickly to changing circumstances, strong, powerful organizations almost never can.
What I Like About Obama
Barack Obama strikes me as a reasonable person, who will make reasonable decisions given his biases.
He’s engaged with scientific and technical communities early in the formation of his policies.
He advocates for more transparency in government.
He seems to recognize the problem of momentum.
He focuses on ideas and policies.
He seems to grasp that what was so despicable about the Iraq War was that it was executed without following sound political processes.
Contrast this with Clinton. She totally misses (or maybe she really doesn’t) that the stupid policies she’s advocated for–from authorizing force in Iraq because, apparently, she thought she was playing a game of chicken, to her original bologna national health care plan, all the way back to her support of the Clipper Chip–would have any negative consequences at all.
And, let’s face it, she and her hubby campaign like people who want power and will do anything to get it.
Another thing in Obama’s favor is his very real dealings with multicultural, multitheological environments. His religious Faith was really a journey and, seems to be, a good parallel to how he makes political decisions. To quote Wikipedia:
Obama writes that he “was not raised in a religious household.” He describes his mother, raised by non-religious parents, as detached from religion, yet “in many ways the most spiritually awakened person that I have ever known.” He describes his Kenyan father as “raised a Muslim,” but a “confirmed atheist” by the time his parents met, and his Indonesian stepfather as “a man who saw religion as not particularly useful.” […] Obama writes: “It was because of these newfound understandings—that religious commitment did not require me to suspend critical thinking, disengage from the battle for economic and social justice, or otherwise retreat from the world that I knew and loved—that I was finally able to walk down the aisle of Trinity United Church of Christ one day and be baptized.”
I am not voting for Obama because I particularly agree with all of his policies.
For example, I do truly believe that nationalized health care will fail, fail utterly, and fail spectacularly. But I also do truly believe that it’s inevitable because we’re tired of mere philosophizing and need first-hand experience. And the alternatives offered by the Republicans aren’t exactly sound.
What Obama seems to offer for this inevitable expansion of government is that he will, again, follow sound political processes and be transparent about its implementation. Which, by the way, gives it a higher chance of succeeding.
And I think, if any politician today could stop the wave, or keep it from knocking over houses when it crashes, or maybe even keep it working well long enough to find a better alternative, it’s Obama.
Of course, it could all be for show. Given that Obama (a) is a politician, (b) is a Democrat, (c) is a candidate for President, and (d) has raised $130M in the last year … well, the chances are good that it’s all an elaborate con. That’s the trouble with power.
Still, I’m voting Tuesday without hesitation. If the hard-core Democrats are smart enough to nominate him, I will, of course, look for more evidence of his reasonableness before November. I think I’ll find plenty.