Stand and deliver -- how can a team become quicker?

Recently I met with an old colleague of mine and we talked about teams, among other things. He asked me what I would do to urge on a team to become quicker and to deliver more. My gut reaction was that this is a thin line to walk. The following delves a bit deeper into the issue.

What not to do

One particular pitfall is to ask the team to deliver more story points. I guess the motivation behind the idea is two-fold:

  1. Delivered Story Points are easy to measure.
  2. The team could work harder and then easily deliver more.

One of those assumptions is generally speaking false and it’s not the measuring point. What I usually see is not people slacking off but people struggling.

But it’s not only that this core assumption is wrong, it gets worse when you consider the likely effects of asking for more story points to deliver. First of all, note that it’s very easy for the team to deliver more story points: The team can simply raise their estimates for each story. They can raise the estimates basically on each estimation, thereby gaming the system and destroying any trust you can have in the estimates. This essentially means you’re loosing the velocity as an indicator what the team can reliably deliver in the future.

The second effect is even worse: If people feel pressure to move more stories to “done”, people will concentrate on delivering the core value — this being the functionality that the client / product owner was asking for. On the flip side, they will stop caring about quality because that is apparently not what is the main driver. As a result, people might stop doing the following things, because they actually cost time, time they can’t spend on finishing the next story:

  • testing thoroughly
  • covering the changes with automated tests
  • cleaning up the legacy code
  • refactoring your first “working” implementation into a maintainable shape

The biggest problem with this is that the effects of all this will usually not affect the current sprint, but will probably cause severe slow down later on, either because of bugs that need to be fixed later or because all the technical debt will make any further changes so much more complicated. Let me repeat this: if you don’t care deeply about the quality of what you build, the effects will slow you down, maybe not today, but certainly tomorrow.

Don’t ask for more points, instead drill deeper to figure out what is holding back the team and focus on fixing that.

Why the team can’t move faster and what to do about it

My friend then came up with the idea that it might be viable to look beyond the usual Scrum process, maybe consider Kanban or something else. My perspective (and reaction to him) is that he was on the wrong track. Yes, Scrum does add some rituals that some people consider to be unnecessary baggage that don’t add value. In my experience that’s either because the team in question is actually already a highly experienced team that delivers very successfully or because the process parts are executed so poorly that they don’t help the team. But usually the problems are elsewhere and simply changing the framework you’re using will not fix those.

Process problems

Let’s be clear: yes, in some cases you actually have a problem in the way work flows through your system. One classical scenario is that all testing work is only started at the end of a sprint, when all the implementation is done or you have too many people working on items in parallel, not being able to finish anything early. You are basically setting yourself up to find bugs too late to be fixed and very often also to have left-over testing-only work that needs to be completed in the next sprint.

If you’re using any method to inspect and adapt (e.g. regular retrospectives), this is one of the things you would expect the team to be able to identify themselves and also to take action on. Some guidance from an experienced Scrum Master can be really valuable here, she could for instance use an activity to identify the value stream through the team. Also, ideas from various frameworks can help, e.g. setting a work-in-progress limit as done in Kanban.

The other classical case of process problems is where the organization is putting up several hurdles in front of the development teams without even recognizing it. This could range from forced use of ineffective tools or low-level machines over tons of bureaucracy to using outdated micro-management practices. These process problems are far from easy to tackle and require usually a lot of help from somebody outside of the development team (e.g. the Scrum Master or some manager).

So, yes, sometimes your process is a problem, but it’s unlikely that it’s a question of Scrum or Kanban (or something else).

Worthless work doesn’t motivate

If your team doesn’t seem to have high spirit to tackle the next job, it’s probably not related to your organization. If your product is really worthless, no one will be motivated to put in any effort. I have a good friend who was literally asked to add piles of sh*t to a game his employer produced. This was a rather short lasting work relationship. I guess such an experience is the exception but nevertheless in some cases people see no value in what they produce.

Let’s assume you team is working on a genuinely useful product. This does not mean that it’s obvious to everybody that the next version or feature serves a useful purpose, too. Sometimes the people which know best why delivering a product to the customer is important are the same which don’t understand the need to explain this to the development team. But this is vital for motivating the good people that are building the product in the first place. Ignoring this often goes along with a mindset of seeing the development team as a feature factory, a black box into which you have to feed specifications and which will spit out the next version of the product on the other side.

The problem is that on the inside of the factory the value of the work remains in the dark, of course. Either feature that needs to be built is just another feature, all different but also all the same. As a result, team members will get the impression that it’s actually not all that important what exactly they are building, outside of meeting the deadline, of course. Why there is a deadline in the first place and what happens if it’s missed or whether the actual customer problem could be solved better with an entirely different solution, is a problem of other people “above my pay grade”. Of course, if the team doesn’t understand the value of what they’re building, how could they care about delivering something with better quality or with higher efficiency?

I’m convinced that the root cause of this issue is that there is no collaboration between the product managers and the development team. Try to live up to the idea that “Business people and developers must work together daily throughout the project”. It’s hard to just throw requirements over the fence without any further explanations, if the guys can ask all sorts of questions every day in the stand-up. Day to day collaboration and just spending a lot of time together (aka “socializing”) will also help with the insight that the developer team consists just of fellow humans with a brain and a genuine interest that their work should matter.

The other answer to this wide-spread problem is to make sure that at the start of a project (release, product increment, sprint etc.) the person responsible for the success and the money explains the vision and the goal that needs to be met. If you can back up this story with customer input or other market research, the better. When explaining and discussing things (the what), keep referring back to the goal and the vision (the why). This will lead to better discussions, better solutions and almost certainly to higher motivation among the team.

Technical debt

“If your code is crap, stickies on the wall won’t help” — Rich Rogers

Stickies on the wall The truth of this quote can’t be overstated in my experience. The more technical debt you accumulate, the slower your team will become as it struggles to understand all that crap. Also, code that is hard to understand will make it easy to introduce new bugs and each new bug means that your team will not work on new features. Bad code will also demotivate your people as they feel like Don Quixote fighting an endless battle against windmills that will constantly throw more bad code into their faces. There is also the broken-window syndrome: if people get the feeling that everything looks awful and is broken already, they can get the feeling that it is alright to put in even more spaghetti code.

“Continuous attention to technical excellence and good design enhances agility” — Agile manifesto

I think it’s a shame that the focus of agile has shifted from methods like XP to frameworks like Scrum or SAFe which don’t have the same emphasis on technical excellence. Don’t get me wrong, I know that a lot of the techniques which were promoted in XP like TDD are now considered state of the art for software development, regardless of the process for work organization you are using. But the thing is that it’s too easy (for people like the Scrum Master) to concentrate on getting stuff “done” and other core pieces of such frameworks and to let the technical excellence slip completely out of focus.

The right action to take is to make people constantly aware that good technical solutions which are maintainable will make life (or at least work) so much easier and nicer. Use a Definition of Done which is focusing on quality and maintainable code. Establish pair programming and code reviews. Make sure you get technical debt items into the backlog and that they are also prioritized appropriately.

Note that this is not about “gold plating” and spending endless hours on polishing code. But honor the Boy Scout rule that you should leave any place you visit a little bit cleaner than before.

Lack of expertise

One related problem is that you might have people in the team which don’t have all the necessary expertise that would allow them to work “faster” or to find high quality solutions. This usually comes in two flavors:

  • lack of general development know-how (e.g. SQL, programming language, design patterns etc.)
  • understanding of the existing code basis

Both are usually problems which will become smaller over time because your beginners will gain more experience in the fundamental technology and also in the code basis on which they are working. Be aware, however, that it’s easy to reach a level where people have learned enough to be dangerous. That’s difficult because they can both produce something that “works” but is not of the required quality. It’s easy to stay on this low-level plateau of understanding when there is no mentoring / coaching beyond this level.

Clueless Cartoon Man Looking At Different TimesThe key idea to solve this problem is to actively invest into the expertise of your team members. One idea is to have dedicated training sessions addressing the required know-how areas. However, getting a theoretical introduction into a topic is usually not enough and should only be the starting point to get hands-on experience. So, I think it is crucial that the overall team understands that the more experienced people can fundamentally help the less experienced people while doing the real work — by patiently answering questions, explaining concepts and reasons, by pair programming and general technical coaching. Note that this isn’t a one-way road (from senior to junior) either: this becomes a lot easier if the inexperienced people are confident that it’s okay to ask for help and to admit failures without having to fear retaliation — which brings us to the next point.

Team conflicts

The agile manifesto prefers “Individuals and interactions over processes and tools”, but it’s so much more easy to focus only on the latter. Especially technical people are often too happy to ignore people problems (maybe that’s why they chose to go into tech in the first place). Team conflicts can come in various different flavors, e.g. some people might not want to work together, maybe there are blame games being played or there are bad rumors being spread behind people’s back. You don’t have to be able to recite the proverbial Five dysfunctions of a team to understand why deep conflicts within a team will be a fundamental blocker on the way to more effective and efficient results. Emotions will get in the way of any discussion, technical or other.

While it is maybe obvious to have this insight into the damaging effects, actually figuring out that there is such a conflict in a team might not be so easy if there is no open fight. Fighting openly is very often avoided by all involved parties, in the name of so-called professionalism. Even if it becomes clear that there is a conflict (or multiple), the team might be very reluctant to actually discuss or resolve it. This is directly related to the conflict, of course, because people need at least some feeling of psychological safety (and ideally also trust) to be able to address problems openly.

I think it’s rare that deep conflicts within a team are resolved without involvement from the outside, regardless of whether the external person is a manager or just another colleague. This takes a lot of patience and even more empathy. An important ingredient here is to make sure that the emotional aspect of the conflict are actually coming to light.

Ironically, the prime directive for retrospectives that is intended to remind people to be respectful can sometimes be a hindrance to openly discuss the conflict. People can misunderstand “believe everybody did the best job they could” that no criticism should be voiced at all, where it’s supposed to enable trust to speak up openly about problems. Activities like Spot the elephant can help here. In my opinion, it is okay when this results in hard discussions and hurt feelings and when emotions become apparent. However, it is the responsibility of the moderator to ensure that these emotions don’t get in the way of a discussion of solutions. Ideally, the involved people should be the ones which come up with the solution. Unfortunately, in the case of deeper personal conflicts, often the involved people have no idea how to separate factual discussions from emotional aspects or have no experience how to give or react to feedback without deepening the conflict. Moderation from some experienced “outsider” is key here. Of course, in some cases, the best resolution might be to change the setup of the team, but this amounts to losing the game.

Keep it simple, stupid

In my experience, the main problem in software development is rarely the technology or the main process, but almost always people and their interactions. Note that also technical debt is coming either from inexperience or from people not collaborating enough to point out the rising quality problems. So, if you think the process is the problem, I would recommend to take a deeper look.

ObTitle: Adam & the Ants, “Stand and Deliver” from “Prince Charming”


Technical debt equals missed quality requirements

Technical debt assumes that you have an existing system and you know already about the areas where the duct tape is becoming thin. Let’s discuss the problem, the conflict around technical debt a little. It’s basically always the same battle: the guys maintaining the system probably have good reasons why they want to pay back on the accumulated technical debt, whereas the product owner believes more functionality is more important.

This is a typical clash of interests between people from different tribes and usually from different departments: the developers report to a technical manager, the product owner to a business guy. The IT manager is typically under pressure to minimize costs, in particular costs for support and maintenance, so he’s interested in paying back on technical debt during projects. The business manager, however, is under pressure to convince new clients with new features. So, that there is a difference in how technical and non-technical people set priorities between technical debt and new features doesn’t come as a surprise: naturally, to the business side this is all about technical issues which they don’t consider to be their concern. It’s only natural that they believe that tackling such technical issues should be solved by the technical folks “somehow”. After all, it were these guys which build the system with all these problems in the first place, and now they want to spend more time and money on this? Although I’ve exaggerated here a little maybe and this line of argument is way too simplistic (given that often debt is accumulated over time), it’s still quite popular. Such product owners seem to believe that they are only responsible for developing new functionality. Corporate culture can contribute to this: some companies have a culture where every new glitter is taken for a star and gets way more attention than the cash cow functionality that keeps existing clients happy.

So some product owners will see technical debt as a separate issue which needs to go on a maintenance budget, a budget that somebody else is responsible for, e.g. the IT department. The problem with that idea is that in a culture where building new features is the only thing that counts, typically no one ever gets around to clean up the technical debt. All what will happen is that you’ll have some people that fix bugs with this maintenance budget. So you don’t actually ever spend the money on technical debt but only on the results of it which of course doesn’t address the root of the technical issues. Also, if you finally do get an approval for a refactoring project, these are really the most boring and horrible types of projects to work on. So it’s not likely to see a lot of happy, motivated people on such projects — and sure enough the most competent people are probably assigned to more valuable projects anyway. As a result and because there is no business pressure on such refactoring projects, as soon as something else comes up, such refactoring initiatives are abandoned or at least down-sized to a minimum. Peter Seibel describes in a recent, lovely written article the need for an Engineering Effectiveness team at Twitter in which he lays out in some details how hard it is to really put up with technical debt.

Let’s drill down on technical debt from the point of view of requirements. Wait a minute, this doesn’t sound right — how could technical debt be a requirement? Obviously, it isn’t, but there is a close relation. Technical debt can come in different flavors: e.g. you really should install a second machine and a load balancer so you’re prepared for failure or we really need to rewrite module foobar.clx to finally get rid of all the spaghetti code that’s slowing us down to oh, we really need to support responsive design, we’re getting more mobile users every day. Now, if you take a look at these three simple examples, they are related to quality requirements: the first is about availability, the second about maintainability and the last about usability. Take a look at the ISO 25000 standard for software product quality page listing all the quality attributes one might want to consider. Technical debt is always tied to some quality that your system should offer, but doesn’t.

A reason for this is that quality attributes are often not made explicit during requirements gathering, regardless of whether you’re following an agile approach or not. The less technical the product owner, the more often they seem to assume that performance, scalability, security and the other -ilities will come out just right by magic. Technicians, on the other hand, know perfectly well that they will not. It’s worse if they don’t that: they will build something that might fulfill the functional requirements but not the non-functional requirements. “Works on my machine” might be fine for a naive developer, but nobody will be happy if takes 5 minutes to load the page for the gazillions of parallel requests in production. There is a reason why software architecture is mostly concerned with quality, if you don’t plan and build for scalability, it’s unlikely you end up with a highly performing system. Take a look at the picture: this bridge wasn’t planned to cope with the amount of traffic it sees nowadays, some time in the 80s somebody decided it would be okay to have three lanes instead of two and didn’t think through the long term consequences.

Rheinbrücke A40 Duisburg-Neuenkamp
The bridge of the highway A40 over the Rhein in Duisburg. It is damaged so bad that trucks are no longer allowed to cross it.
Picture made by kaʁstn Disk/Catstitched by Daniel Schwen, licensed under CC BY-SA 3.0 de over [Wikimedia Commons]( https://commons.wikimedia.org/wiki/File:Rheinbruecke_Neuenkamp_pano.jpg#/media/File:Rheinbruecke_Neuenkamp_pano.jpg)

Of course, technical debt can also accumulate over time, one might rightly point out: sure, one machine was enough for the requirements at the start, but nobody followed up on the increase in users and nobody cared (or had time) to clean up the code in module foobar.clx back then (nor in the following six years of quick bug fixes and minor one-off patches that have grown on it like leeches). This is a sign that nobody actively had an eye on how the world in and around the systems changes and to take action early on. For code quality, Uncle Bob points to the Boy Scout rule which says that you should always clean up (regardless of who messed it up) — another way of trying to ease maintainability by paying back on technical debt every day.

The over-aching point here is that you, as a technician and you, too, as a product owner need to think about the quality requirements of your system and to do that over the entire life cycle. What was a fitting solution at a time might turn into technical debt over time. This means means your system now doesn’t hold up to the quality requirements you and your clients needs and no matter what was the cause for it, you’re better of fixing it now than accumulate even more technical debt.

Isaac Sacolick describes in his article on How to get an agile product owner to pay for technical debt how to address the common problem of technical debt heads on, mostly from a managers perspective via process and by making people (mostly the technical lead) responsible. However, I think it makes a lot more sense to try to ensure a common understanding between the developers and the product owner. One way to go about this is to use quality scenarios: as a member of the dev team, ask your product owner how many users the system needs to handle. Ask her if she thinks it’s okay if somebody might find a security issue earlier than you. Or if she thinks it’s okay when a seemingly small change will take three weeks because the code is an intangible mess that nobody understands since Dieter left? These sort of questions hopefully open up to discussions based on business value, ROI and cost of delay. You are then using the language product owners understand and you might also learn something along the line (no we expect so many users to do X in the future, so investing in module Z doesn’t make sense).

Now, granted that doesn’t always work. An old colleague of mine always denounced the old system he worked on as being exactly the way it was ordered. Isaac’s article might offer some advice here, of which the most important one is probably that paying back technical debt is way easier if you have a CIO or someone with similar power supporting you.


Dependency inversion in Clojure

The problem

I was recently reading a nice German book on Effective Software Archictecture by Gernot Starke and stumbled upon a discussion of the dependency inversion principle, which got me thinking. Gernot Starke first discusses the problem with an allusion to traditional procedural programming (translation mine):

Classical designs for procedural languages show a very characteristic structure of dependencies. As (the following figure) illustrates, these dependencies typically start from a central point, for instance the main program or the central module of an application. At this high level, these modules typically implement abstract processes. However, they are directly depending on the implementation of the concrete units, i.e. the very functions. These dependencies are causing big problems in practice. They inhibit changing the implementation of concrete functions without causing impacts on the overall system.

Classical dependencies in procedural systems He then goes on to introduce the idea of programming against abstractions and introduces the idea of the dependency inversion principle, first coined in Bob Martin’s DIP article (see also another thorough discussion in Brett Schucherts article on DIP). Basically, the idea is that the integrating process refers only to abstractions (i.e. interfaces) which are then implemented by concrete elements (classes), cf. the next figure.

Integrate with abstractions When I take a look at some of my recent Clojure code or at some older code I’ve written in Common Lisp, I immediately recognize dependencies that correspond to those in a classical procedural system. Let’s go for an example and take a look at one specific function in kata 4, data munging:

(ns kata4-data-munging.core
    (:require [kata4-data-munging.parse :refer [parse-day]]
              [clojure.java.io :as 'io]))

(defn find-lowest-temperature
    "Return day in weatherfile with the smallest temperature spread"
    [weatherfile]
    (with-open [rdr (io/reader weatherfile)]
         (loop [lines (line-seq rdr) minday 0 minspread 0]
        (if (empty? lines)
            minday
            (let [{mnt :MnT mxt :MxT curday :day} (parse-day (first lines)) ;<-- dependency!
              curspread (when (and mnt mxt) (- mxt mnt))]
            (if (and curday curspread
                  (or (= minspread 0)
                  (< curspread minspread)))
               (recur (next lines) curday curspread)
               (recur (next lines) minday minspread)))))))

The dependency here is on the concrete implementation of parse-day, you can basically ignore the rest for the argument here. Given that this was a small coding kata, this is not unreasonable (and in the course of the kata, the code changes to be more general), but the issues here are obvious:

  • if we would like to parse a weather-file with a different structure, we have to change find-lowest-temperature to call out to a different function,
  • if the result of the new function differs, again we have to change the implementation of find-lowest-temperature,
  • we also have to change the namespace declaration, i.e. we probably want to require a different module.

Clojure’s built-in solutions

The application of the dependency inversion principle is typically shown in the context of object-oriented programming languages, like Java where you use interfaces and classes implementing those interfaces for breaking the dependency on concrete implementations, cf. the figure above again. But as we’ll see the principle can be applied independently of object-orientation. I’ll discuss higher-order functions, protocols and multimethods as potential solutions.

Higher order functions

For starters and probably painfully obvious is to make use of the fact that Clojure treats functions as first-class objects and supports higher-order functions. This simply means that we can pass the parsing function as an argument to find-lowest-temperature.

(defn find-lowest-temperature
    "Return day in weatherfile with the smallest temperature spread"
    [weatherfile parsefn] ; <-- function as parameter
    (with-open [rdr (io/reader weatherfile)]
         (loop [lines (line-seq rdr) minday 0 minspread 0]
        (if (empty? lines)
            minday
            (let [{mnt :MnT mxt :MxT curday :day}  (parsefn (first lines))
              curspread (when (and mnt mxt) (- mxt mnt))]
            (if (and curday curspread
                  (or (= minspread 0)
                  (< curspread minspread)))
               (recur (next lines) curday curspread)
               (recur (next lines) minday minspread)))))))

This way, we can simply call (find-lowest-temperature "myweatherfile" parse-day) and freely substitute whatever file format and accompanying parse function we need. What does this buy us?

  • We no longer have to modify find-lowest-temperature when we want to use a different parse-day function.
  • The namespace containing find-lowest-temperature also no longer requires the (namespace containing the) parse function.

But there is also a down-side: find-lowest-temperature assumes that all parsing functions it will get fed adhere to a signature that is entirely implicit: parsefn needs to take exactly one line and needs to return a map with given key-names. Higher-order functions don’t provide a solution for this per-se, so in order to solve the implicit signature issue we need to look elsewhere. This is nothing Clojure specific: Assuming you’ve passed in an object either as a method parameter or via Setter-Methods or Constructor-Injection (cf. dependency injection), Python’s or Ruby’s duck-typing basically works the same way: the caller of a method simply assumes that the callee offers a method with the right signature. It is the responsibility of the caller (of find-lowest-temperature) to provide a matching function for parse-fn.

However, this actually amounts to just move the problem from one level to the next: now some other level has to decide which concrete parse function needs to be used. This next level will have again the exact same problems: it will depend on both concrete implementations of find-lowest-temperature and parse-day (or any other parse function). If you think this through, it’s obvious that in general at one point or another, you have code that determines which function to call and which parameters to use. The question is only if we can use abstractions or whether we have to use concrete implementations. We’ll return to this issue, that now at some other level you need to handle the problem, later.


Page 1 of 1, totaling 3 entries