Agile - 2

Last week, I visited this years Goto Zürich 2013, which is a two-day conference with tutorials wrapped around it. Topic areas range from lean start-up over technology to a leaders track. I spent most of my first day on the so-called leaders track as here most talks revolved around adopting agile and lean methods, but also went to some more technical talks in other tracks, too. I’ll summarize most of the talks and my impression below. Btw., the slides to most of the talks can be found on the conference website, more precisely on the schedule overview for the two days.

The conference started out with a keynote by Scott Ambler on his work on Disciplined agile delivery which provides a framework on where decisions are needed when implementing agile methods in a larger setup, drawing from many different methods like Scrum, XP, Kanban, the Scaled Agile Framework and many others. The first part of his talk was mainly concerned with the question if agile holds up to its promise of delivering ‘better’, which he tries to answer with regular surveys. These seem to confirm that team size, location, complexity and methods used do have an impact. Not surprisingly, small team size are more successful than larger, co-located teams better than distributed ones and simple projects are much more likely to succeed than complex ones. And of course, there are still projects using waterfall-like approaches that succeed, while the difference between an iterative and an agile approach are rather minimal. Still, however, the number of project failures are always high, even for simple projects there are more failures than one might expect.

The first talk here was about Spinning by Ralph Westphal, which is well-known e.g. for his association with the Clean Code Developer initiative. His premise was that today’s business reality is a constant change of priority which is at conflict with assuming bigger time boxes during which developers can focus on some goal. His answer is that we should ‘seize the day’ (as Dan North might have put it) and deliver ‘value’ daily. It’s important that this value should be something worthwhile to the customer and the customer should be able to give feedback daily as well. I’m wondering whether it is really always possible to accomplish this, e.g. fixing a bug might require analysis well beyond a day. But even if you might not always hold up to the idea, it might still be a worthwhile guideline for organizing work. Another question which I don’t have an obvious answer for is the question who should be in charge to decide what should be worked on on any given day? Ralph required a thorough triage to be carried out to avoid wasting time, but it’s unclear whether this job belongs e.g. in the hands of the product owner.

Dominik Maximi spoke next about ‘Hostile waters’, e.g. how company culture might influence your chances and approaches to introducing agile in an organization. He made the important point that every company culture exists because it is (or was in the past) successful to work like that. This needs to be respected when you want to change something fundamentally. He then gave a nice overview of the Schneider model on how to classify company culture. Non-representative survey results indicate that ‘agile’ has a similar characteristics to ‘collaborative’ or ‘cultivation culture’, but doesn’t fit in so nicely with a ‘competence’ culture or ‘control’ (no surprises here). Changing the mindset of a company might take up to 7-10 years. Dominik finally discussed John P. Kotter’s work on change steps to implement agile.

The need for estimates is a fact of (project management) life, whether you’re using a traditional approach or agile ones. The latter try to do away with any exact numbers and instead use anything from t-shirt sizes over gummi bears to story points and treat the idea of estimates as a tool that is mainly intended for the team. However, as to be seen in this article posted on the ScrumAlliance website on story points versus task hours the matter of concrete time estimates comes up every now and then — there are customers and managers that want it and you have to figure out how your estimate / velocity is related to the team’s capacity for any given sprint. TL;DR:

For me, story point is high-level estimation of complexity made before sprint planning.[…]The task-hour estimation, on the other hand, is a low-level estimation[…]should be done during sprint planning for highest possible accuracy.

But, just like every other project manager, I made the experience that there is no such thing as “highest possible accuracy”, regardless of the number of ‘points’ you’re estimating (three-point estimates, anybody? How does that fit in with agile methods?) What does somewhat “work” (for varying degrees of “somehat”) is measuring how much work is ‘roughly’ possible in an iteration (aka velocity in Scrum) and estimating the ‘entire’ efforts according to the law of big numbers, but this is exactly not task-hour estimation. I’ve also worked with task hour estimates during the sprint planning as well, but I wouldn’t rely on the results. Exchanging estimates between developers (aka planning poker) has the main effect of quickly leading to a common understanding of the tasks that need to be done, which in my experience has much more value than any estimate. The insight that numeric estimates are not all that useful (to say the least) is also discussed in this highly opionated article on stop using story points — this is not about going back to estimating in hours or days, but about teams reaching a level of competence, expertise and confidence in their respective context where estimates are just waste.

However, reaching this transcendental level might be hard to reach for many teams, due to lack of experience or because of other more context dependent reasins. Joachim Seibert discusses in his nice article for the “ObjectSpektrum” magazine (5/12) “Agile estimating in teams” (behind a paywall, unfortunately and only in German, sorry), that if you need to come up with the big number for budgeting, you should strive to look at the entire picture. I.e. estimating the overall effort is more important than trying to completely estimate everything down to the tiniest detail — which isn’t possible anyway. Seibert lists two key ingredients for that: estimating a reference story which you stick to throughout the project and comparing the relative complexity of requirements, essentially sorting them by ‘size’. This is easy to forget with planning poker, but is at the very heart of the two other techniques that Seibert discusses: estimation game (by Steve Bockman, apparently) and magic estimation (by Boris Gloger, AFAICT). Both focus on building a common understanding of the relative complexity of the features to be build and are much simpler to apply to lots of stories. If you add velocity to the picture, i.e. you actively measure how much you can build and you can reach a stable project setup (compare this nice sweet spot for agile methods by Philippe Kruchten), you might be able to end up with something useful from all the effort you put into estimation.

One aspect I think is important for a Scrum Master or Project Manager is to make sure that your team doesn’t go on a trip to Vienna (if that term doesn’t ring a bell, search for “Tom DeMarco peopleware”). Quite contrary to popular management belief, I think in general it’s not okay if “occasionally” somebody on the team “puts in some extra work”. There is a reason why many agile methodologies insist on keeping a sustainable pace. Besides all of the very good reasons for making sure your team members stay healthy (see this Burnout story as a negative example), there is also a management point to it: your understanding of what the team is capable of (in terms of results/effort, aka velocity) decreases substantially if you have to take “heroic behaviour” into account. It’s particular bad when you don’t see the connection between reached goals and involved effort, i.e. when team members just move their card from “working” to “done” late in the evening without making clear that it involved five hours more than initially estimated.

Heroic behaviour just can’t be counted on, because nobody will be able to keep it up over a substantial amount of time (that’s the very definition of not being substainable). It’s highly understandable that project members after having committed to some goal can be tempted to go out of their way to reach it. What team members might miss is that “heroic behaviour” can only have an influence on the “time” aspect of the magic triangle of “time, budget and quality”. Extra effort is just that: effort. Hence, it comes with a cost, with the cost it just takes to reach the goal. I’ve also seen that there is a misunderstanding of the term “commitment”. It’s not an unconditional promise of “I can do that task with the effort I think it takes”, there is also the implicit condition of “I understand correctly what the task involves and there is no other external negative influence” (e.g. the urgent bug that needs to be looked at, or the lack of sleep during three days of the week due to the kids being sick at home).

Commitment to a particular goal might at times conflict with taking responsibility for the project as a whole. As a general rule of thumb it’s nearly always much more important to think about the entire project / the big picture than about a small aspect of it. There is the exceptional situation that needs exceptional reaction and maybe exceptional effort. But it’s important to treat it like an exceptional situation. And for these exceptional situations it’s vital that they get treated like a mini project: they should have a clear purpose and have fixed start and end dates. Plus, they should come with a compensation. Scrum Masters and project managers alike should communicate clearly that exceptions are exceptions, not the rule. And team members should clearly communicate that it takes what it takes. When it comes to professional work, follow the 501 manifesto (in case you don’t directly understand the “501” part like I did: it’s not about jeans, but about leaving at 5:01pm).

ObTitle: Morrissey, from “Ringleader of the tormentors”

In a recent meeting, a colleague of mine mentioned that we wanted to use agile development this time with that new project and that I would provide some insight as an ‘agile development expert’ (not that this would be a term I would use). This in turn brought me some curious looks and a pretty general question during the discussion: what is the benefit of an agile development model from a business perspective anyway?

If I remember correctly, my answer went in the direction that with an agile approach you can change your mind about what you really need/want to implement throughout the project, if you learn something ‘new’ about what is really required. The second important aspect is that you get quicker feedback which allows for more influence in case things don’t turn out right. After the meeting I started pondering the question a little, as I had the impression that my answer wasn’t that convincing, so what you find below is a more detailed written elaboration of it.

Basically, what “you can change your mind throughout the project” boils down to is that agile development gives you more options to make use of opportunities coming up along the way/throughout the time you need to implement the project. The most important idea here is that you don’t make decisions too early, because that limits your options. For an example, let’s assume that one feature we specify is that we want to have a ‘Facebook like’-button on the order page. Now, if we’re following the waterfall style ‘implementation follows after specification has been completed’ development model, you would specify that and design the page with the button and at some time in the future, you hopefully get your Facebook button. But maybe during the hypothetical nine months from now in which development happens, what if Germany’s privacy laws enforce new severe restrictions on the use of such buttons and, btw., Google+ is now the new hip site you have to support? You have to go through a (with larger features typically quite costly) change process, redo the specification, etc. This essentially makes the cost you had in writing the specification as well as for the design etc. a double problem: you didn’t get out any ROI, obviously, but additionally time has passed while writing specs, doing design etc. in which you didn’t really work on any results for the customer (lost opportunity: gather new clients with at least some new features).

So, you would have been better off with having the idea that at some point throughout the project you want ‘social media integration for marketing reasons’ and delaying what to do exactly until you’re really there (from a priority point of view) where this is a feature that you want implemented — i.e. you do the specification when your planning that feature to get implemented in the next development phase (which is typically two to four weeks with an agile development team). The idea is to take small, well defined steps that can be taken fast and starting out with the most important/valuable ones first, rather than to spend a lot of time on some ‘complete specification’ which withers fast throughout the projects implementation time.

The problematic point here is obviously, that it’s pretty difficult to say when exactly is the right point in time to make a decision. In general, there is no ‘right point’ — also often called the last responsible moment after which some opportunity is lost (I think the phrase was coined by Mary and Tom Poppendieck, see the excerpt from ‘Lean software development’) — but multiple points in time, depending on many aspects. Influencing aspects are things like business opportunities (e.g. a new feature nobody else has at that time) or reducing risks (projects risks) (cf. Alistair Coburn reconsidering the least responsible moment). The best answer I currently have seen when to make a decision is that it should be taken when you have ‘enough knowledge’, i.e. when you have carefully considered your options and evaluated them (cf. real options). Even thinking about options alone is typically a highly valuable undertaking in itself, because all to often people just assume that the ‘obvious way’ is the best one.

So, the general answer is probably along the way of: for the business side, agile development has the benefit that it provides more flexibility and more influence throughout the project phase while at the same time (possibly) generating real value/revenue more early. There is, of course, more to it that also has appeal from a business perspective, but that’s probably the main point. Feel free to add different opionions.

If you just stumbled over this blog entry searching for what the heck all that buzzing about agile methods amounts to, I may have unfortunate news for you: An agile approach to development might for a lot of people all over the world, but may be not for you. For starters, take a look at the agile manifesto and its principles. These have quite a lot of implications, directly translating into presuppositions about you, if you wanted to participate in an agile development project.

Now, perhaps you’re the well communicating, team oriented developer type who happens to work in a surrounding which you respect and general feel well motivated. But real life experience with your typical geek or ex-geek-turned-professional-developer suggests that having a hard time with direct communication, face-to-face, happening very often with those pesky guys and girls you don’t really like is a more likely scenario (“Eeek, those business and marketing people” anybody?). Now, you might say, baah, I know some communication is a necessary evil of professional software development, but maybe I can get away with my old habits of trading documents (specs, bug reports, etc.) against direct communication, since that is the trusted old way and what do you mean by face-to-face communication being more effective? Everybody hates those horrible meetings eating up all your time, right?

The point is that if you’re wanting to go agile you should better soon adapt your view or else you won’t see any of the promised benefits of all of those agile processes. Agile development is not only about using test driven design, timeboxing, iterations and releasing often. More than anything else, agility is about how people effectively interact with each other as quickly and direct as possible in order to come to solutions. To me, this basically boils down mainly to three issues: adaptability to (changing) situations, an open mind toward communication and taking responsibility for the project at hand.

Being lazy, sticking bone-headed to “trusted paths”, not seeking confrontation when it’s needed, avoiding communication because of being too busy “doing” is the anti-pattern to agility. You could keep a product and sprint backlog, present new features every other week to your customer and even use continuous integration and still would be muddling in pseudo agile, doomed to fail water. What I would suggest is that you should revisit the agile manifesto and think about what those principles might imply for you and your current behaviour. Let’s try an example: If you find that you might end up pointing fingers at your customer (or product owner to use the terminology from Scrum) because he won’t behave as demanded in the agile manifesto, you should reconsider if that fits in with what the agile manifesto demands of you: you wouldn’t do the project any good. You would build up frontiers where there should be a single team including the target of your pointed finger. This is not to say that you shouldn’t point out the defect, no communication about problems is another all too familiar reason for frustration and lack of commitment. What is necessary is direct communication and finding a way that works for all people involved.

Let me sum up this posting: if you’re interested in agile approaches to software development, that’s fine. If you don’t feel comfortable about what this might imply for you, that’s fine too. But it might also tell you that agile development approaches might not work for you.

Update: Irionically, I stumbled via Infoq about the one essential agile ingrediment which says all about it much nicer than I ever could have done, only that Mark Schumann shows the confidence that agile works for the most part whereas I hold the more pessimistic view that it’s not going to work for some, but for exactly the same reasons.

ObTitle: Hermann Hesse, “Steppenwolf”