Make it so! Decision making in software architecture

I find it interesting in discussions with developers that many have a bad picture of the job of a software architect. When thinking back to my first encounters with a software architect, who was performing his document reviewing job in a dedicated architecture department, I can totally relate, but then again it is 2021 and things should have changed [1]. After all, most developers have a very keen interest that the architecture of the system they are working on is good, but still some developers do not seem to be aware how much their work is impacting the architecture (and vice versa). “Every interesting software-intensive system has an architecture. While some of these architectures are intentional, most appear to be accidental”, as stated in Grady Booch’s article on “Accidental Architecture”. Some people now seem to believe that accidental architecture is the new norm, as “the best architecture, requirements and design emerge from self-organizing teams” (from the principle in the Agile Manifesto). In contrast, in organizations which have dedicated software architects, some people seem to think that it is the architect’s responsibility to ensure that there is a well informed design and to make the hard decisions as necessary. These hard decisions would probably be about “the important stuff (whatever that is)” that is called out in Martin Fowler’s article “Who needs an architect”.

But what does that mean for the daily work, should developers send all questions to the architect for her to make decisions and should architects sit in their ivory tower pondering them and ordering decisions by importance? Not in my view of the world. I’m a big fan of the role model that Fowler describes as Architectus Oryzus or acts as primus inter pares in Georg Hohpe’s article “Agile and Architecture: Friend, not Foe” in which the architect is closely working with the developers and who tries to identify important topics and make sure they are getting addressed at the right time.

Sketch image showing "A Design thinking Workshop", by Jose Berengueres, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons.
A Design thinking Workshop
Image by Jose Berengueres, CC BY-SA 4.0, via Wikimedia Commons.

Stefan Toth has a fantastic (German only) book on agile processes for software archictecture Vorgehensmuster für Software Architektur which describes patterns how to make this happen in daily practice. The core point of the book is that architecture should be a shared responsibility. Everybody should try to raise and address important concerns — whether there is a dedicated architect role or not is a separate issue. Toth lists patterns like identifying quality scenarios, listing architecture topics in the backlog, ad-hoc architecture meetings, common decision making and testing for architectural concerns (and many more) many of which are very team centric. The book is so great because it shows that it is easy to make architecture work a natural aspect of the development workflow and not some high ceremony.

In my experience, technical discussions are often concerned with architectural topics, so you might not even need to bring people together for formal architecture meetings — nevertheless technical design review meetings can be beneficial if they involve people that can and do share new perspectives but who would otherwise not be working so closely with the team. Similarly, if your team already adds technical debt and other non-functional items into the backlog, you probably are already following two of the recommended practices.

The role of the architect is then not so much the one of a decision maker but more the one of the person taking responsibility that architecture aspects are considered by the team. In bigger organizations this implies that the architect should be working closely with the development team but not only. While ideally the developers are also closely collaborating with any other important stakeholder (product owner, devops / operations, other dev teams), it is quite vital for an architect to understand the different (and potentially conflicting) quality needs of these parties. Again, probably even better than having e.g. 1:1 meetings between architect and security officer, so that the architect can point out operational constraints to the development team, would be if the architect tries to make sure that a security expert is involved in design and requirement discussions.

Hippo with tongue stuck out (via Pixabay)
Highest paid person’s opinion.
Image taken from Pixabay

However, the job of an architect is not only consisting of organizing meetings with the right people. In the end, intentional architecture is in need of decisions. “Common decision making”, as listed above, means two things to me: first, people who are affected by a decision (e.g. the developer who has to implement the result) should be directly involved in the decision making process. In teams that do not have a dedicated architect this seems naturally to be the case, but it is worth pointing out that this can easily degrade to single programmer decisions. The other anti-pattern here would be that the architect, likely to be the highest paid person in the room, makes a HiPPO decision or specifies a design while at the same time not being close enough to the problem and / or the existing code base. [2] The effect here can be devastating if the decision turns out to be wrong: not only will people think that the architect did not listen to the team but also that the architect might not have the necessary competence. This can result in serious psychological safety and trust issues. A good working approach to avoid such problems is running small experiments (spikes) to validate options.

The second aspect is that a decision is made in the first place, which is even more important. The idea of decision in the last responsible moment notwithstanding, I find it surprising how hard it can be for organizations / a group of people to take decisions. It is important for an architect to be able to guide a group towards making a decision but also to be comfortable to make a decision themself, if the group cannot find an agreement. The biggest issue here are the trade-offs of potential solutions: e.g., one solution might have benefit A (e.g. scalability) while being weaker in B (e.g. ease of use) and another solution has opposite characteristics. Often different people will have different opinions whether A or B is more important in the selection process. It is very helpful for teams to come to an understanding (not agreement) about these different weighs that people give to the characteristics and then hopefully, the architect can explain why she thinks that one argument is more important than the other. Coming to a decision is more important than convincing everybody, though: aim for solutions that everybody in the team can agree with, even if they would still prefer a different one. Also, it is a good idea to document decisions in a light weight fashion (e.g. in a small Architecture Decision Record).

In case that during implementation the developers run into obstacles, it is quite likely that the developers will actually figure out a different solution than the one decided. This is another case where the “architect as lonely decision maker” can lead to a lot of unnecessary friction. In practice, the choice of a different approach is not a problem as long as it is handled in a similar responsible way, where the developers take a look at the implications and trade-offs of their choice and involve others (and, please, update that ADR).

Footnotes:

[1] I am obviously not talking about so-called Enterprise Architects who are mostly concerned with governing how many systems at a company interact to provide the most business value. Instead the focus here is on software architecture for a single system (which, of course, is likely to interact with many others).

[2] Sometimes the opposite scenario might be true: the team is consisting mostly of people which do not have enough expertise, whereas the architect has long experience with the code base. In this situation, the architect can be helping the team by sketching a design approach and sharing his knowledge. Still, there should be room for questions and further ideas from the team.


Angela Merkel, HiPPOs and decision making

So Angela Merkel yesterday withdrew the decision to add another non-workday before Easter (cf. deutschland.de video of Angela Merkel’s press conference), apologizing and taking full responsibility for the half-baked idea. Many people paid her respect for this and so do I. Some thoughts on this, though:

First of all, this half-baked idea was the result of a meeting going on for far too long. As some MP said, they first heard of the idea at 23:45 in the evening, after the discussions had been running for many hours. As most knowledge workers would know, a typical effect of working too hard for too long is that the quality of your outcomes is diminishing. Too often, reflecting back on the result of yesterday’s overnight session will reveal that your fabulous idea might actually be totally off.

Second, apparently no experts were around which would have an idea of what it would entail to make this additional day-off a reality. This looks a lot like a decision based on “HiPPO”, the Highest Paid Person’s Opinion. Generally speaking, results are better if experts prepare options with effort / benefit arguments. And usually, it’s better to postpone a decision to “do it this way” until you know that it actually can be done that way.

Last but not least I find it a pity that the behavior of Angela Merkel, to openly admit a mistake, is apparently exceptional and surprising to many people. Admitting mistakes and taking responsibility should be the norm for any leader and not the exception. As a leader, you want to understand the problems your team is facing and what mistakes people are making. How else could you improve your results? But if you are not leading by example and are not open about your mistakes, it is likely that your people might have the impression that it is only okay to talk about the great accomplishments. This in turn means that everybody is afraid to make any mistake, trying to reduce risks at all costs, which will kill any positive attitude and atmosphere. Good luck trying to achieve great results in a context like that.

tagesschau.de article on Angela Merkel’s press conference to withdraw the day off before Easter


Color is the key -- configuring my Dygma Raise keyboard

In 2020, I finally came to the point where I was ready to invest some money into my health. So I bought a standing desk and a new keyboard, a Dygma Raise, which is an ergonomic split keyboard with mechanical keys. The main reason to buy the Dygma instead of some other ergonomic or mechanical keyboard was that I wanted to be able to keep my arms straightly positioned when using the keyboard — with a regular keyboard I always need to move my arms a little bit inwardly in front of my chest, which leads to a body position where I’m not keeping my back straight which contributes to back pain. As you can see from the image below, I move the two parts considerably apart from each other, to the extent the rather short connection cables from the keyboard to the so-called Neuron allow it. Fortunately these are just USB-C to USB-C cables which means I can just buy two longer cables when I want to (didn’t get the round tuit yet).

Picture of my dygma raise

Configuring the Dygma Raise

The Dygma Raise is a highly configurable keyboard. Not only can you order various layouts (ANSI, ISO) or colors, you can also order different key switches and also change keys as you see fit — yes, I’m talking about the hardware keys here. The my raise page gives a nice overview. I ordered a German layout and Cherry Brown keys (cf. the nice switch overview), which gives a nice tactile feedback without being too loud. I did not fiddle with the keys themselves so far but would like to talk about the various key layers and their configuration via Bazecor, which works on Linux, Mac and Windows. I only tried to use Bazecor on Linux (Debian 10), where they provide an AppImage which so far has been working good (minor point: the user has to be in the dialout group, otherwise you need to start the application as root).

Initial impressions

My initial impression, coming from a pretty regular Cherry G86, was: “Great haptic feeling, but oh my, this will take time to get used to using it.” During my initial attempts to use the keyboard I recognized how often I apparently had been looking at the keyboard before: when I moved the two halves apart and not look on the keyboard, I would utterly mistype. I overcame this by really practizing to type blindly with the two parts put together, so that the typing was more akin to what I was used to. Over time, I got better at this and can now type well without having to look at the split halves.

The biggest other problem for me was that the default key layout has no cursor keys on Layer-0 at all, the cursors are on Layer 1 and keys W,A,S,D. To get to Layer-1, I could either use the transinient Shift to Layer-1 key (the left middle lower thumb key) or the persistent Lock to Layer-1 (the right middle lower thumb key). So, what was always a single keystroke was now requiring two key presses in a totally different area of the keyboard. Add to this the use of all sorts of modifiers that I’m constantly using, e.g. C-M-left to switch screens and now I have to press another key to switch the layer? Not nice. I similar missed Page-Up and Page-Down, which were not even configured for Layer-1. And then, what do I use instead of the media keys I used on the Cherry? That should be easy to configure, right.

Goals

It became very clear very quickly to me that I have to adjust my typing behavior. However, I use currently two machines in parallel and I only have one Dygma Raise. Obviously, I need to configure replacements for at least some of the keys that are missing in comparison to my old keyboard.

The other thought was that I use a lot of key combinations in various programs I use and here I have ten layers to configure, so surely I should be able to ease some of my typing? I.e., in Emacs instead of htting C-M-SPC C-w to kill the current S-expression, it would be nice to just hit a shorter key combination.

And finally, the Dygma Raise has colors like a rainbow, so …

  • keep layer 0 configuration close to a normal PC-105 layout
  • use layer 1 for missing keys, especially for the media play/stop key
  • use layer 1 and others for shortcuts / special needs
  • setup consistent and meaningful color usage

After some time, I settled to use three layers like this:

Layer 0

As stated in the goals, layer 0 is setup to be pretty normal, see below. E.g, I set all four “space” keys to yield space, so this is like a single big space key. Also, different from the pains that is described in Kari Martilla’s blog about his dygma raise, there are no changes to the parenthesis whatsoever, although they are equally problematically placed on the German layout as on the Nordic layout.

To address the missing cursors, I configured the lower row of thumb keys for the cursors which makes their use very easy, as I only need to hit a single key that is also very easy to reach. For switching virtual desktops in LXDE/Openbox, I use C-left or C-right and this is even easier now, as I can now move to right with Right-Control, too, where before my right hand would have to wander off to the cursor block.

Layer 0 configuration

One key difference is the Escape key: on a PC-105 with German layout that is where you would find the caret (^), but I decided to keep the Escape key bound to it. The caret is to be found on layer 1. And then there are the Dygma and the FN key(caps): they are bound to Shift-to layer 1 and 2, respectively.

Note also the color usage which clearly demarks group boundaries, e.g. between the space and alt keys.

Layer 1

Layer 1 mostly holds the keys which I need often but for which there is no room on Layer 0. I.e., the caret is on the Escape key and the function keys (which I actually only very rarely use) are on the number keys. For the remaining movement type keys, my line of thought went like this: Q and A go to Home and End, W and S to Page Up and Down, because that positioning resembles the key ordering of these keys on a regular PC104/105 layout. Delete and Insert, however, are needed more often and hence can be reached via the two lower left thumb keys.

Layer 1 configuration

I set the media keys (next, previous, louder, quieter) to N,P,+,-, respectively, which makes it easy to memoize. The start/stop toggle is bound to FN, so that (on layer 0) I can simply hit the dygma+FN combination.

You can also see the Menu, Led cycle and Move to Layer 1 are configured on this layer but I don’t use these keys much. I’m currently experimenting with putting the parenthesis additionally on layer-1 on K,L,Ö,Ä to see if that makes hitting the parenthesis [,],(,) that I need for Clojure programming most often better than on the regular keys, but I’m not sure yet how good that will work.

A lot of keys are configured to produce nothing, grouped in the default color of the layer, whereas the configured keys pick up the colors from layer 0 plus the dark blue for the media keys.

Layer 2

With Bazecor versions beyond 0.22, it is now also possible to configure macros, i.e. a sequence of keys. Initially that wasn’t working for me: you have to upgrade the firmware of the Dygma to enable it. That’s the main purpose I started to configure a layer 2 for. Currently I have configured only two such macros: Q (i.e. Fn-Q when I’m in layer 0) triggers a sequence of keys that, when pressed in a lisp mode in Emacs, will select the current S-expression and indent it. It is bound to Q as this is somewhat similar to hitting M-Q will do (formatting a paragraph). W will trigger a key sequence that in Emacs will select the current S-expression and kill (cut) it, similar to C-w.

After I’ve set this up, I thought about doing this with a regular Emacs keyboard macro or Elisp function instead and then invoke these with the keys. This would offer two benefits: I could assign this just to the modes there this combination makes sense and I have the nice side-effect that invoking these functions can be done independent of the Dygma being in use. As it is, if I now hit Fn-Q while not being in Emacs, the application in use will receive the defined key macro sequence and who knows what happens then. Then again, the entire point is to make use of the Dygma to shorten the amount of keys to be pressed.

I also set up ‘-’ to generate C-_, which will trigger an undo in Emacs, thereby on my German keyboard saving me one key stroke (Fn-<-> instead of C-Shift-<->).

Layer 2 configuration

I also configured the Dygma key to toggle media start/stop, so that it doesn’t matter if I shifted to Layer 1 or 2. And, finally, there is an experimental alternative configuration of some movement keys (Home, End, PgUp/Down) on the thumb keys.

Conclusion

The Dygma Raise is an expensive keyboard, no doubt. Did it help with my back pain? Not so much, unfortunately. But it is an amazing keyboard that allows to be configured in lots of ways. Would I recommend it? I surely would. Oh, and did I mention that it comes with a case to be carried around? Ah, keyboard nerds galore.


Who Am I and If So How Many? Using multiple Firefox profiles.

So, I’ve changed my Firefox setup quite a bit over the last months. I now work with multiple profiles, e.g. one for online banking, one for social media use and one for development purposes and one for “default” use (i.e. pretty much else). I’m quite happy with the experience.

Usually I have all three profiles running in parallel, showing on different virtual desktops of my linux box. I.e., on desktop 2, I usually have my development environment open, so I’ll open the “development” browser there, desktop 4 runs all the applications for interacting with other people (mail etc.), so “social” goes there and the default one will show on virtual desktop 5.

Why would you do this?

One direct effect is that the amount of open tabs in any given browser window is a lot smaller and also better grouped than before. I.e., any web page I need for my current development project I will only open in my “development” profile. I will only open my bank’s page in the banking profile, so this browser window will never show anything else.

The other main benefit is that I can have profile specific configurations. E.g., I nail down my “default” profile with NoScript which is not really useful on my “development” profile, whereas I don’t need e.g. the React Dev tools on the “social” or “default” profile.

Dedicated profiles can also help with security, e.g., using a dedicated profile can lower the attack surface for online banking: When you don’t browse to other sites with the same browser/profile, any XSS/CSRF issue on these “other” sites for sure can’t affect your online banking connection.

The profile I use exclusively for online banking is also highly locked down and in addition uses a different theme, so that it is visually obvious that I’m working with this profile.

Get me started, please

To start using multiple profiles, you have to run Firefox with the -P switch, which will start the Profile Manager that allows you to create new profiles, delete profiles etc. Alternatively, if you have Firefox already running, browsing to about:profiles will also allow you to manage your profiles.

For a while, I just started the non-default browsers manually over the command line, by just opening an xterm and running firefox --no-remote -P social &. But I finally created some additional local .desktop files (cf. the Arch wiki page on xdg desktop files), so I can start Firefox from the desktop. I.e., I added a file $HOME/.local/share/applications/socialbrowser-usercreated.desktop with the following content:

[Desktop Entry]
Name=Social Firefox
Comment=Browse the World Wide Web
Comment[de]=Im Internet surfen
Exec=/usr/lib/firefox-esr/firefox-esr --no-remote -P social %u
Terminal=false
X-MultipleArgs=false
Type=Application
Icon=firefox-esr
Categories=Network;WebBrowser;
MimeType=text/html;text/xml;application/xhtml+xml;application/xml;application/vnd.mozilla.xul+xml;application/rss+xml;application/rdf+xml;image/gif;image/jpeg;image/png;x-scheme-handler/http;x-scheme-handler/https;
StartupWMClass=Firefox-social
StartupNotify=true

This will then create a menu entry in the “Internet” submenu in my application starter menu in my desktop environment (because that’s where the given categories will create entries).

Any drawbacks?

One annoying thing is the profile selection that Firefox pops up when you don’t specify a profile on the command line. If you select the option “Use the selected profile without asking at startup”, then it will not be easy anymore to use a different profile — the only way then is really to use -P again.

This default profile selection becomes a problem especially when you want to open a link from a different application (eg. from your mail program), because then you can’t decide which of the running browsers/profiles will open the link, it will always try to use the default one. I have seen varying behavior what Firefox does when you don’t select a default profile: sometimes it just picks one running profile successfully, but I’ve also seen it opening the profile selection dialog again. In that case, if you select a profile that is already in use, Firefox will handle it like an attempt to open up the profile a second time, resulting in an error. My current workaround to this particular issue is to set the default browser to Chromium via xdg-settings set default-web-browser chromium.desktop.

The other hassle working with multiple profiles is bookmark management, as I want some bookmarks only local to one profile but most should be shared. I can use Pocket for the shared ones, of course. However, I often just copy&paste the URL manually to the “default” browser which serves as the main bookmark keeper. I really should move away from this completely and instead use the bookmark extension of my Nextcloud installation.

Overall, for me the benefits clearly outweigh any drawbacks.


File download with ClojureScript

As I couldn’t find a recipe on how to provide some data from a ClojureScript application for download, here’s how. If you know how to do this in JavaScript already and if you’ve done any CLJS-JavaScript interop, there’s nothing new for you to learn here, as this is a pretty straight-forward translation of how to use the Blob API and clicking a temporary link in JavaScript,

(defn file-blob [datamap mimetype]
  (js/Blob. [(with-out-str (pp/pprint datamap))] {"type" mimetype}))

(defn link-for-blob [blob filename]
  (doto (.createElement js/document "a")
    (set! -download filename)
    (set! -href (.createObjectURL js/URL blob))))

(defn click-and-remove-link [link]
  (let [click-remove-callback
    (fn []
      (.dispatchEvent link (js/MouseEvent. "click"))
      (.removeChild (.-body js/document) link))]
    (.requestAnimationFrame js/window click-remove-callback)))

(defn add-link [link]
  (.appendChild (.-body js/document) link))

(defn download-data [data filename mimetype]
  (-> data
       (file-blob mimetype)
       (link-for-blob filename)
       add-link
       click-and-remove-link))

(defn export-data []
  (download-data (:data @some-state) "exported-data.txt" "text/plain"))

I’ve tried to break it down into pretty self-explaining pieces, but here is a bit of explanation: export-data would be used as an on-click handler on some UI element and would expect to gather the data to be exported in some way. Here, we’re just assuming the data is already stored in some state-atom and is a map. file-blob is pretty-printing the data somewhat, declaring the content to the given MIME type (e.g. text/plain) and then returning the newly created blob. Of course, you might want to change the pretty printing or the MIME type, depending on your data.

We’re not doing anything fancy with the created Blob object, we simply hand it over to URL.createObjectURL whose result we use as the href attribute of a newly created anchor element. Setting the download attribute will tell the browser not to navigate to the URL. After this we simply add this new link to the DOM and then execute the download by dispatching a MouseEvent on said link. This actually triggers the download, i.e. the browser will open up a file save dialog with the suggested name, so the only thing left to do is to clean up the link from the DOM.


Stand and deliver -- how can a team become quicker?

Recently I met with an old colleague of mine and we talked about teams, among other things. He asked me what I would do to urge on a team to become quicker and to deliver more. My gut reaction was that this is a thin line to walk. The following delves a bit deeper into the issue.

What not to do

One particular pitfall is to ask the team to deliver more story points. I guess the motivation behind the idea is two-fold:

  1. Delivered Story Points are easy to measure.
  2. The team could work harder and then easily deliver more.

One of those assumptions is generally speaking false and it’s not the measuring point. What I usually see is not people slacking off but people struggling.

But it’s not only that this core assumption is wrong, it gets worse when you consider the likely effects of asking for more story points to deliver. First of all, note that it’s very easy for the team to deliver more story points: The team can simply raise their estimates for each story. They can raise the estimates basically on each estimation, thereby gaming the system and destroying any trust you can have in the estimates. This essentially means you’re loosing the velocity as an indicator what the team can reliably deliver in the future.

The second effect is even worse: If people feel pressure to move more stories to “done”, people will concentrate on delivering the core value — this being the functionality that the client / product owner was asking for. On the flip side, they will stop caring about quality because that is apparently not what is the main driver. As a result, people might stop doing the following things, because they actually cost time, time they can’t spend on finishing the next story:

  • testing thoroughly
  • covering the changes with automated tests
  • cleaning up the legacy code
  • refactoring your first “working” implementation into a maintainable shape

The biggest problem with this is that the effects of all this will usually not affect the current sprint, but will probably cause severe slow down later on, either because of bugs that need to be fixed later or because all the technical debt will make any further changes so much more complicated. Let me repeat this: if you don’t care deeply about the quality of what you build, the effects will slow you down, maybe not today, but certainly tomorrow.

Don’t ask for more points, instead drill deeper to figure out what is holding back the team and focus on fixing that.

Why the team can’t move faster and what to do about it

My friend then came up with the idea that it might be viable to look beyond the usual Scrum process, maybe consider Kanban or something else. My perspective (and reaction to him) is that he was on the wrong track. Yes, Scrum does add some rituals that some people consider to be unnecessary baggage that don’t add value. In my experience that’s either because the team in question is actually already a highly experienced team that delivers very successfully or because the process parts are executed so poorly that they don’t help the team. But usually the problems are elsewhere and simply changing the framework you’re using will not fix those.

Process problems

Let’s be clear: yes, in some cases you actually have a problem in the way work flows through your system. One classical scenario is that all testing work is only started at the end of a sprint, when all the implementation is done or you have too many people working on items in parallel, not being able to finish anything early. You are basically setting yourself up to find bugs too late to be fixed and very often also to have left-over testing-only work that needs to be completed in the next sprint.

If you’re using any method to inspect and adapt (e.g. regular retrospectives), this is one of the things you would expect the team to be able to identify themselves and also to take action on. Some guidance from an experienced Scrum Master can be really valuable here, she could for instance use an activity to identify the value stream through the team. Also, ideas from various frameworks can help, e.g. setting a work-in-progress limit as done in Kanban.

The other classical case of process problems is where the organization is putting up several hurdles in front of the development teams without even recognizing it. This could range from forced use of ineffective tools or low-level machines over tons of bureaucracy to using outdated micro-management practices. These process problems are far from easy to tackle and require usually a lot of help from somebody outside of the development team (e.g. the Scrum Master or some manager).

So, yes, sometimes your process is a problem, but it’s unlikely that it’s a question of Scrum or Kanban (or something else).

Worthless work doesn’t motivate

If your team doesn’t seem to have high spirit to tackle the next job, it’s probably not related to your organization. If your product is really worthless, no one will be motivated to put in any effort. I have a good friend who was literally asked to add piles of sh*t to a game his employer produced. This was a rather short lasting work relationship. I guess such an experience is the exception but nevertheless in some cases people see no value in what they produce.

Let’s assume you team is working on a genuinely useful product. This does not mean that it’s obvious to everybody that the next version or feature serves a useful purpose, too. Sometimes the people which know best why delivering a product to the customer is important are the same which don’t understand the need to explain this to the development team. But this is vital for motivating the good people that are building the product in the first place. Ignoring this often goes along with a mindset of seeing the development team as a feature factory, a black box into which you have to feed specifications and which will spit out the next version of the product on the other side.

The problem is that on the inside of the factory the value of the work remains in the dark, of course. Either feature that needs to be built is just another feature, all different but also all the same. As a result, team members will get the impression that it’s actually not all that important what exactly they are building, outside of meeting the deadline, of course. Why there is a deadline in the first place and what happens if it’s missed or whether the actual customer problem could be solved better with an entirely different solution, is a problem of other people “above my pay grade”. Of course, if the team doesn’t understand the value of what they’re building, how could they care about delivering something with better quality or with higher efficiency?

I’m convinced that the root cause of this issue is that there is no collaboration between the product managers and the development team. Try to live up to the idea that “Business people and developers must work together daily throughout the project”. It’s hard to just throw requirements over the fence without any further explanations, if the guys can ask all sorts of questions every day in the stand-up. Day to day collaboration and just spending a lot of time together (aka “socializing”) will also help with the insight that the developer team consists just of fellow humans with a brain and a genuine interest that their work should matter.

The other answer to this wide-spread problem is to make sure that at the start of a project (release, product increment, sprint etc.) the person responsible for the success and the money explains the vision and the goal that needs to be met. If you can back up this story with customer input or other market research, the better. When explaining and discussing things (the what), keep referring back to the goal and the vision (the why). This will lead to better discussions, better solutions and almost certainly to higher motivation among the team.

Technical debt

“If your code is crap, stickies on the wall won’t help” — Rich Rogers

Stickies on the wall The truth of this quote can’t be overstated in my experience. The more technical debt you accumulate, the slower your team will become as it struggles to understand all that crap. Also, code that is hard to understand will make it easy to introduce new bugs and each new bug means that your team will not work on new features. Bad code will also demotivate your people as they feel like Don Quixote fighting an endless battle against windmills that will constantly throw more bad code into their faces. There is also the broken-window syndrome: if people get the feeling that everything looks awful and is broken already, they can get the feeling that it is alright to put in even more spaghetti code.

“Continuous attention to technical excellence and good design enhances agility” — Agile manifesto

I think it’s a shame that the focus of agile has shifted from methods like XP to frameworks like Scrum or SAFe which don’t have the same emphasis on technical excellence. Don’t get me wrong, I know that a lot of the techniques which were promoted in XP like TDD are now considered state of the art for software development, regardless of the process for work organization you are using. But the thing is that it’s too easy (for people like the Scrum Master) to concentrate on getting stuff “done” and other core pieces of such frameworks and to let the technical excellence slip completely out of focus.

The right action to take is to make people constantly aware that good technical solutions which are maintainable will make life (or at least work) so much easier and nicer. Use a Definition of Done which is focusing on quality and maintainable code. Establish pair programming and code reviews. Make sure you get technical debt items into the backlog and that they are also prioritized appropriately.

Note that this is not about “gold plating” and spending endless hours on polishing code. But honor the Boy Scout rule that you should leave any place you visit a little bit cleaner than before.

Lack of expertise

One related problem is that you might have people in the team which don’t have all the necessary expertise that would allow them to work “faster” or to find high quality solutions. This usually comes in two flavors:

  • lack of general development know-how (e.g. SQL, programming language, design patterns etc.)
  • understanding of the existing code basis

Both are usually problems which will become smaller over time because your beginners will gain more experience in the fundamental technology and also in the code basis on which they are working. Be aware, however, that it’s easy to reach a level where people have learned enough to be dangerous. That’s difficult because they can both produce something that “works” but is not of the required quality. It’s easy to stay on this low-level plateau of understanding when there is no mentoring / coaching beyond this level.

Clueless Cartoon Man Looking At Different TimesThe key idea to solve this problem is to actively invest into the expertise of your team members. One idea is to have dedicated training sessions addressing the required know-how areas. However, getting a theoretical introduction into a topic is usually not enough and should only be the starting point to get hands-on experience. So, I think it is crucial that the overall team understands that the more experienced people can fundamentally help the less experienced people while doing the real work — by patiently answering questions, explaining concepts and reasons, by pair programming and general technical coaching. Note that this isn’t a one-way road (from senior to junior) either: this becomes a lot easier if the inexperienced people are confident that it’s okay to ask for help and to admit failures without having to fear retaliation — which brings us to the next point.

Team conflicts

The agile manifesto prefers “Individuals and interactions over processes and tools”, but it’s so much more easy to focus only on the latter. Especially technical people are often too happy to ignore people problems (maybe that’s why they chose to go into tech in the first place). Team conflicts can come in various different flavors, e.g. some people might not want to work together, maybe there are blame games being played or there are bad rumors being spread behind people’s back. You don’t have to be able to recite the proverbial Five dysfunctions of a team to understand why deep conflicts within a team will be a fundamental blocker on the way to more effective and efficient results. Emotions will get in the way of any discussion, technical or other.

While it is maybe obvious to have this insight into the damaging effects, actually figuring out that there is such a conflict in a team might not be so easy if there is no open fight. Fighting openly is very often avoided by all involved parties, in the name of so-called professionalism. Even if it becomes clear that there is a conflict (or multiple), the team might be very reluctant to actually discuss or resolve it. This is directly related to the conflict, of course, because people need at least some feeling of psychological safety (and ideally also trust) to be able to address problems openly.

I think it’s rare that deep conflicts within a team are resolved without involvement from the outside, regardless of whether the external person is a manager or just another colleague. This takes a lot of patience and even more empathy. An important ingredient here is to make sure that the emotional aspect of the conflict are actually coming to light.

Ironically, the prime directive for retrospectives that is intended to remind people to be respectful can sometimes be a hindrance to openly discuss the conflict. People can misunderstand “believe everybody did the best job they could” that no criticism should be voiced at all, where it’s supposed to enable trust to speak up openly about problems. Activities like Spot the elephant can help here. In my opinion, it is okay when this results in hard discussions and hurt feelings and when emotions become apparent. However, it is the responsibility of the moderator to ensure that these emotions don’t get in the way of a discussion of solutions. Ideally, the involved people should be the ones which come up with the solution. Unfortunately, in the case of deeper personal conflicts, often the involved people have no idea how to separate factual discussions from emotional aspects or have no experience how to give or react to feedback without deepening the conflict. Moderation from some experienced “outsider” is key here. Of course, in some cases, the best resolution might be to change the setup of the team, but this amounts to losing the game.

Keep it simple, stupid

In my experience, the main problem in software development is rarely the technology or the main process, but almost always people and their interactions. Note that also technical debt is coming either from inexperience or from people not collaborating enough to point out the rising quality problems. So, if you think the process is the problem, I would recommend to take a deeper look.

ObTitle: Adam & the Ants, “Stand and Deliver” from “Prince Charming”


Threadripping under Debian

Getting a Threadripper machine to work under Debian

After a long long time of more than eight years my old machine started showing hardware problems: first the power supply failed and had to be replaced. Next the CMOS battery died. It became clear that finally a replacement would be in order. When the first reports about AMDs Ryzen family came out and in particular with the Threadripper tests, I became interested. In the end, I waited until the Threadripper 2950X was available before ordering a custom-built machine from a local vendor. Here is the basic hardware setup:

  • CPU: Threadripper 2950X (32 cores)
  • Mainboard: MSI 399X Carbon
  • Memory: 32 GB
  • GPU: Nvidia 1060
  • SSD: Samsung 970 Evo NVM 1TB

This post will describe what I had to do to on the software / linux side to bring the machine to a usable state.

Basic installation

As a Debian user of old and knowing that the latest Debian stable (Stretch) would not provide a recent enough kernel, I installed Debian Buster. This went more or less flawlessly. However, I can’t help but note that the installation of lvm and disk encryption (ie. lvm2 with lukssetup) provided by the Debian installer is really bad. It hasn’t improved one bit since ages: the automatic partitioning will come up with a crazy sizing scheme and there is absolutely zero useful support when you go manual. It’s easy to end up with an installation which looks like it succeeded but rebooting will end up on the rescue console because mounting the root volume group fails.

The only other noteworthy thing is that I directly installed the proprietary Nvidia drivers because the free nouveau driver doesn’t really support any of the “newer” and more advanced 10X0 cards from Nvidia. As I am a LXDE user, that was of course the desktop environment I installed. However, I installed LXQt and KDE along with it, to also learn about their current state, but went quickly back to LXDE — LXQt is supposed to be the successor of LXDE but it’s currently not really in a comparable state.

I also installed Gnome briefly, only to discover that it’s trying to use Wayland which apparently has problems with the Nvidia drivers. I couldn’t make it work and as I was never a big Gnome fan anyway, I simply threw out Gnome again.

system-udev crashes and kernel compilation

With the system up and running, the first problem I noted was system-udev constantly crashing. This also prevented suspend/hibernate from working. A longer internet search finally revealed this system-udev bug report in Red Hat’s bugzilla. Apparently “the BIOS/firmware is advertising it supports SEV, when in fact it doesn’t” where SEV is AMD’s secure encrypted virtualization technology. If the kernel config option CONFIG_CRYPTO_DEV_SP_PSP is set to yes then the kernel would use it and apparently most distribution provided kernels are setting it — the one from Debian Buster (4.18.0-2 at the time) surely does.

There are apparently three ways to fix this issue:

  • wait until the BIOS/firmware is fixed to no longer provide the wrong information,
  • wait for a newer kernel to provide a workaround,
  • or compile the current kernel with CONFIG_CRYPTO_DEV_SP_PSP set to no.

Apparently, since 4.19-rc5 the kernel has a workaround for the issue, but at the time of writing Buster and Sid only have 4.18. So, off to compile a new kernel without that setting. This turned out to have become a lot more complicated since the last time I had to do this on a Debian system (which was still using make-kpkg at the time, so yes, that’s several years I didn’t had the need to compile a kernel). It also doesn’t help that quite a bit of documentation out there is outdated — the Debian Kernel handbook seems to be the proper documentation.

Unfortunately, it’s still easy to get it wrong. I ran into issues with certificates (why exactly you would need certificates to compile a kernel is beyond me, my wild guess is it’s related to “secure” boot) or my changes to the configuration were overwritten during the process and other issues. In the end, the following process “works for me”:

    # cf. chapter  4.2.3, generate the setup for amd64
    make -f debian/rules.gen setup_amd64_none_amd64

    # fire up the config dialog, now enable RCU-BOOST and disable PSP
    make -C debian/build/build_amd64_none_amd64 xconfig

    # build the kernel
    make -f debian/rules.gen binary-arch_amd64_none_amd64

The only problem with this is that the generated kernel will have the same numbering as the official package. So, a newer minor version of the Debian package can overwrite your manually build package. I haven’t really looked into this yet, but will update this section once I do (and yes, the process described in section 4.3 of said manual didn’t work for me).

Btw., don’t expect kernel compilation to be a fast process. Apparently, the kernel configuration that is used out-of-box for Debian compiles half the world and consequently this takes ages even on such a high-end machine.

The newly compiled kernel fixes the systemd-udev crashes and then also suspend / hibernate worked. However, when I triggered hibernate from the LXDE logout dialog, the machine wouldn’t power-off. Another long round of searching the web, including reading the source to lxsession-logout.c, revealed the solution to disable upower via systemctl disable upower.

Random hangs

During my first attempt to compile a kernel, my system just hang. Completely, not even the magic sysreq worked anymore. That’s apparently another known issue, but there are no good solutions. The attempt to set RCU_NOCB_CPU und RCU_BOOST is one such attempt (cf. the soft hang discussion in the AMD forum, but this didn’t really help me (cf. also the random soft lock discussion on the kernel bugtracker). However, the also linked ZenStates github repo contains a Python script which disables the C6 state.

Again, the forum suggests that the issue might be fixed with newer BIOS versions, but my mainboard has the mentioned AGESA version and the issue still occurred. Disabling C6 state per the script fixes the problem, but results in higher energy consumption and is hence not exactly a perfect solution either. If you want to run zenstates automatically, there is also a systemd template for zenstate. Note that this needs some additional tweaking (which I didn’t get around to yet) to run modprobe msr.

PCI errors

One other thing I noted in the logs were re-occuring PCI errors. There are a number of suggested fixes, cf. this PCI error answer on askubuntu to use pci=nomsi or pci=noaer or this other PCI error suggestion on unixstackexchange to use pci=nommconf. For me, pci=noaer hides the errors successfully and for the moment that’s good enough (read: I haven’t investigated whether the other suggestions would actually fix the issue also).

Virtualization

The latest thing I ran into was that Virtualbox was refusing to start a virtual machine, claiming the AMD feature would be disabled in the BIOS. This turned out to be the case, SVM was disabled. Actually, I couldn’t easily find the option in the first place, I had to use the search in the Bios.

Two things I haven’t tried out yet

The two things I haven’t used yet are the goodies the motherboard provides over the one I originally wanted: Bluetooth and Wlan. I did install the intel firmware to support both features, but can’t really say anything more about it yet.

Conclusion

I don’t have a conclusion just yet. It’s clear this new machine is a lot faster than my old box, but given the Core2Duo cpu is also quite long in the teeth now and that the old one didn’t had a SSD, that’s to be expected. Although this article has gotten longer then I would have hoped for, overall the machine is running quite well (with me running a testing version of Debian which is usually really badly supported). In particular, I can’t say I ran into any issues with the peripherals so far. I guess the machine will have to show it’s power over the next months and hopefully there are some more fixes to BIOS/firmware and the kernel as needed.


More than the sum of its parts -- splitting requirements

In the last article on the right size of a user story, I discussed some aspects that contribute to finding an answer that works in your context: clarity, functionality and effort. In this post, I want to discuss the question on how you can move from a big feature to smaller stories. The assumption here is that we already understand the requirements “good enough” (i.e. maybe captured in a use case document), so we have reached already a quite clear picture of what needs to be done.

Špalek na štípání
By Chmee2 (Own work) [GFDL, CC-BY-SA-3.0 or (CC BY-SA 2.5-2.0-1.0 ], via Wikimedia Commons

There is quite bit of literature on these questions, with the book “Agile software requirements” from Dean Leffingwell being one particular source of input (the SAFe page on stories lists an overview). Being asked, what would you recommend, the usual consultant answer would be “it depends”, but I think it is safe to come up with some rules of thumbs.

0. Don’t split

The bottom line for story splitting is that you should avoid it if you can. Having a lot of very small stories will come at a price — loss of overview is an important one, and if taken too far, it can also become hard to reason about the business value of a story. Formulating the “… in order to” part of the classical user story template (“As a <role>, I want <action> in order to reach <my goal>”) is often a good sanity check: if you have a hard time finding the wording for “my goal” is usual a sign that you’ve split up the requirement too much. There is one classical mistake that can easily result in this situation: if you break down a requirement by components or systems, it’s highly likely you will end up with problems to formulate a reasonable business goal — most probably you split off an implementation task, cf. below.

Assuming the main reason is to fit the work into one iteration (or one week of the iteration, cf. above), if your story is already “small enough” in terms of effort, then there is not much point in going for even smaller stories. You will run into the occasional discussion that a story should ideally be able to finish (“move to done”) within one day, but it should be up to the team to decide whether pushing for this really brings any benefits like additional helpful feedback. More often than not, the team might just try to game numbers, e.g. the number of stories they can finish. There is an obvious tension here between the benefits of small stories (delivering working code early, enabling quick feedback) and unnecessary overhead (splitting in a meaningful way, documenting, managing and tracking the bigger amount of stories), that needs to be balanced according to the situation at hand, but try hard to avoid splitting into too tiny details.

The other question is, do you need to split the requirement now? If the requirement does not need to be elaborated now, because it is not planned for the current or the next iteration, there is no need to split it now — potentially this requirement could change over time anyway, so preliminarily driving for details might actually cause more problems than help. For instance, when a bigger feature involves multiple workflow steps and you split the feature into stories, where each implements one step at a time, you might have a hard time in your backlog tying these pieces / steps together, if you want to revamp these steps in its entirety.

1. Do the minimal thing first

If you think you need to split down a requirement, try to identify the “minimal viable functionality” as the first story. This is the minimum functionality that the system needs to provide that brings some (potentially rather minimal) benefit to the user. This functionality becomes the first story, whereas all other aspects of the requirements are addressed (as extensions) in separate stories.

Ballpoint-pen-parts
Which parts provide the minimal usable functionality?
Foto assumed to be by Pavel Krok~commonswiki under [CC BY-SA 2.5], via [Wikimedia Commons]

As an example, let’s again think about user preferences for some system: the minimum useful functionality for a user consists probably of setting the preferences, saving, loading and applying them. Simple counting these steps leads you to the observation that this alone consists of at least four different functionalities. However, exporting my preferences to a file and importing them on a separate system is certainly not “minimal”. Depending on the intended users, even providing an inline editor to set the preferences might not be part of the minimum functionality. These functionalities could become separate user stories. The drawback of using this approach is that the resulting stories are not truly independent any more: you cannot prioritize exporting and importing preferences to be done before setting, saving and loading them.

Now, of course, this minimal viable functionality might still consist of a lot of work which might still be “too big” (in implementation effort, not fitting into one iteration). Here all the ideas from Dean Leffingwell are applicable, in particular trying to split by persona or business rule variations, data entry methods or data variations or workflow steps. In our example, setting preferences and saving as well as loading and applying preferences could be two different workflow steps (and hence different stories). However, this approach would probably result in stories that are too small according to rule of thumb 0 above. When splitting by workflow steps, try to have a minimal complete workflow in one story or at least plan to have this minimal workflow implemented in one iteration.

2. Optimize later

A good guiding principle in programming is to avoid premature optimization. While the principle talks mostly about optimizing performance of computations, we can also use it as a rule of thumbs for splitting requirements. The idea is to to take the simplest approach that could possibly work. This has a rather broad applicability in terms of technical aspects (architecture), but also usability and other so-called quality or non-functional requirements. After implementing this first rough version, use the feedback to check if it is “good enough” — e.g., whether performance, usability or security requirements are met already. If not, optimize these “-ilities” later on.

Some people might find this suggestion offensive. Security, for instance, is one quality that many claim cannot be designed into a system as an afterthought. But this is also not what I’m suggesting. Instead, I’m suggesting to formulate these “-ilities” requirements separately to isolate the functional core of the big requirement and to implement the optimizations later. This is quite independent of the need to design them — whether you design them upfront (as this might be part of your constraints in a “Definiton of ready”) or within a Sprint is a different question, but of course you need to think about the required qualities that the functionality needs to have. The idea here is simple: If your story is too big, put the proverbial lipstick on the pig in the next round. If you are afraid that your product owner will not care about security and will lower the priority of implementing these security stories to the point that they will never be implemented, your product has a bigger problem anyway (and probably security problems already) or the usage scenario that the product owner has in mind might be different from your expectation. Probably in both cases spelling out the required quality (and the accompanying work and effort) in a separate story could even be helpful to spawn a discussion.

That being said, breaking down your stories by functionality should be preferred. This is especially true if your stakeholders have some expectation of these qualities — if you miss out on these qualities, you’ll likely receive feedback to that effect. At least, you’ll be able to point to another story then to address it. Again, finding a good balance is important, but also not always easy.

3. Avoid splitting by technical components

Fat Slice pepperoni pizza slice
Who would prefer the naked pizza crust over a slice of the real thing?
By BrokenSphere (Own work) [GFDL or CC BY 3.0 ], via [Wikimedia Commons]

Technical user stories that are intended to cover work specific to some component are an anti-pattern. Typically they are brought up by people which are still stuck in the old trap to split work according to responsibility for features vs. components, cf. the discussion of features and components in SAFe. Don’t split by technical components unless the component can deliver business value / user functionality on its own.

Typically you cannot test some functionality in a reasonable way for some component on its own (e.g. the UI alone, not connected to the database). Hence, you will also not be able to get any feedback for the implementation of the story. E.g., saving and loading your preferences from the database is certainly a reasonable technical task, but how would your stakeholder test this without the UI being implemented? How would your product owner find out that there is a minor problem in the saved data if the only way to access it requires sending a web service request or proficiency in your programming language of choice? So, even though you might have finished the work on the component completely, you will not get the quick feedback that we would like to enable with small stories. And, of course, the component alone will also not be delivered as a result of your iteration, so it’s also not adding any business value at this point either.

This rule of thumb also can be debated. There are situations where one component can provide value already — e.g. a CMS system where you extend some functionality on the authoring environment first, so the editors can easily start using it. Even if the readers cannot yet see the result of the new features, some business value might already come from easing the work the editors. This case should not be confused with a different line of reasoning: For the UX designer, for instance, it might make a lot of sense to provide feedback on a UI preview that is not connected to the database. Your DB admin might already point out problems with the schema etc. However, this is inter-team feedback then, not customer feedback. In the end, the measure of progress in agile projects is delivering working software to the customer — and from a user perspective, both of these examples fall short, as essential functionality is missing. Instead, go back to rule 1 and try to identify the minimal complete slice of functionality that delivers value to the customer.

Task splitting and Spikes

There is, however, of course also a time when it is exactly the right thing to do to break down a story into technical tasks that need to be finished: this happens when the story is ready and due to be worked on given its priority, typically as part of a Sprint Planning 2 meeting in a Scrum setup. Especially for engineers it is easy to fall into the trap to confuse story splitting (as requirement work upfront) with task splitting (organization of concrete work within a sprint). A focus on technological considerations (and often also personal priorities) can quickly lead to splitting work such that it is developer-friendly, but does not deliver results to customers quickly. Let’s discuss some aspects of how task splitting influences work organization.

Stating the obvious, splitting technical work into tasks requires technical understanding, both of the system in which the requirement is implemented and of the technologies involved. But it is also not always the case that the team understands the system or the technology well enough. In this case, the team can decide to spend some time on research work to figure out what exactly needs to be done. Don’t define this as a “research implementation” task and plan to complete the entire story, you likely have no idea how much work it takes to complete everything anyway. Instead treat the research work separately, this is usually called a spike. As the goal of a spike is to learn something and not everything, a strict timebox should be defined (usually not exceeding one or two days), after which the spike is called of, regardless of how much insight was generated. Spikes usually are hence not estimated in story points, but reduce the capacity of the team, so you should also try to not have more than or two spikes per iteration.

Felling snags on fire line around the Coquille CCC camp, Siskiyou National Forest (3226072285)
Work in parallel to get it done
By OSU Special Collections & Archives, via [Wikimedia Commons}

Splitting a story into tasks follows different rules: for instance, splitting by components is usually a good idea, especially if your team does not consist only of full-stack developers. Also, even with full-stack developers, you need to start somewhere, which good software architecture practices would demand to be clarifying interfaces between the various components. Which brings us to one big issue of task splitting: Identifying dependencies between tasks. These are usually directly following from the organization of the components. Of course, as with user stories, you should strive to avoid dependencies in the first place (and use dependency injection and other mechanisms to decouple your dependencies). This is not only good engineering practice, but also allows to parallelize the resulting work (i.e. have multiple people work on the same story). Teams sometimes split their work such that only one developer is actively working on a story that consists of multiple tasks. This usually leads to the situation that many stories are started but they all take a long while to be completed. Instead try to distribute work between multiple people: the frontend ninja can start his JS foo, while the backend expert whacks the database in parallel, modifying the previously defined contract as required during the work. If the team can work on multiple tasks of a story in parallel, then this does not only encourage collaboration and team spirit, it will also ensure that the highest priority work items are finished first.

But the tasks to complete a story usually also involves other work besides coding, e.g. UX work or manual exploratory testing. More often than not, these are not happening as agile “on the spot” as other work, instead it’s a quite common anti-pattern to treat an iteration like a mini waterfall: UX (of everything) happens at the start, then development, testing (of everything) happens at the end. This can lead to a lot of unneccessary work done upfront (UX for stories which the team actually doesn’t get around to) or too late (testing starts so late it cannot be completed in the Sprint). Again, try to do the work only when it is necessary and try to do more stuff in parallel.


Requirement anorexia -- does size matter?

One of the recurrent puzzles with regard to requirements management that I keep running into is the question what would be the “right” size of a requirement. It often starts with some people coming up with very tiny and very specific requirements and then some very huge, completely fuzzy ones to go along with them. Then they have heared or read somewhere that in agile setups requirements should be small and wonder what this means and how to get there.

Cartoon by Geek&Poke on Epics
Advanced Scrum
Cartoon by Oliver Widder, licensed under CC BY 3.0 from [Geek&Poke]( http://geek-and-poke.com/geekandpoke/2016/11/6/advanced-scrum)

These huge requirements are often called “epics”: An epic is a requirement that is either felt to be too “large” or is too “fuzzy” and one way to avoid never-ending dramas is to split these into tinier user stories. Actually, here we already have two very distinct aspects: a requirement engineering question of how to move from unclear feature ideas to concrete detailed requirements on the one hand and a requirement management question on how to organize requirements such that they are appropriately prepared for implementation. I discussed the requirement engineering part in an article on agile conception already, so I’ll focus on the latter part here. This size question is actually two-fold: What is a reasonable small story? And how can we break down a big feature into several smaller stories? I’ll address the first one below, the next one in an upcoming article on splitting requirements.

Why small stories?

Let’s first explore the reasons to go for smaller stories. As mentioned above, there is often a felt need to come up with “small” user stories, whatever “small” might mean. This is often even part of making a requirement “ready” for the team to start working on it (cf. also how a felt need for “small” stories is a hint for immature teams).

The main reason behind moving from “big” feature requirements to “small” user stories typically boils down to the team wanting to ensure that the work they take on can be finished within one iteration. From the agile point of view, that’s a good idea: We want to deliver value in terms of working software as quickly as possible. But even if your delivery is not exactly continuously and also doesn’t always happen at the end of the iteration, there is a lot of value in using small stories: developers, UX, product owner, stakeholders should give and take feedback on the completed work to ensure the development is on the right track (in terms of “Did we make it right?” and validation “Did we make the right thing?”). The earlier we learn about problems, the better, so we can make easy adjustments. Arguably small stories can be implemented more quickly and thereby help getting the necessary feedbacks quickly. Also, in case adjustments need to be made, changing a small story is hopefully easy (instead of changing a big requirement). Small user stories also help planning: for one thing, it is usually far easier to come up with an accurate and precise estimate if you’ve broken down a big problem (“epic”) into tiny pieces.

What is a reasonable small story?

This one has an easy answer: any requirement that the team feels can be surely finished within one sprint is reasonably small.

Does size equal effort?

Unfortunately, this answer isn’t particularly useful. It is simply taking the outlined main reason for small stories as a minimal acceptance criteria. If we take a step back, the size of requirement seems here to be directly bound to the effort it takes to implement it. It looks like a continuum: from “no implementation effort” to “takes the entire team the entire sprint to implement”.

But a story that takes the team the entire sprint to implement is almost certainly still too big. Think about it from a risk perspective: let’s assume your estimate is just a bit wrong (as in “too low”), e.g. you discover some previously unknown technical problem during the implementation. In this case you will not be able to finish the story within the sprint — from a work management perspective, you clearly want to avoid this situation, as you’ll run into all sorts of unwanted discussions (from relevant ones like when you get feedback to irrelevant management ones like what can be done to help the team deliver on time or to come up with better estimates).

Burn up charts
Effect of story size on burn up chart

Some teams take a very harsh stance at this and demand a “ready” story should not take longer than one day (of implementation and testing). I don’t believe in such strict rules but think a good upper bound is “no longer than one week”. With story point estimation this recommendation might look strange, but the simple idea is that if you know your team delivers ca. 20 points per two-weeks iteration, a 13 point story is probably too big, whereas an 8 point story could be small enough.

Of course, you can use the same reasoning of risk and project management to demand even smaller stories: e.g., with a 20 point velocity of a team, and three stories with 8 story points each, how would you plan your next sprint? Commit to two stories (summing up to 16 points) and leave one as a stretch goal? If you start working on the stretch goal but are not able to finish it, your efforts and results will not be visible (for the stretch goal). Instead, if you could break down the stretch goal into a 3 and a 5 point story, you might be able to commit to more points. Also, your burn-up chart might look a lot smoother with smaller stories than with big stories, cf. figure 2. But beware: what we really want is quick feedback and delivery of valuable increments. If smaller stories help you with this, great, if not, you’re likely just gaming numbers or making management happy (cf. also Johanna Rothman on the value of burndown and burnup) .

Consider functionality, too

However, the size of a requirement could also be addressed differently. Boris Gloger (e.g. in his German book on agile estimation) argues that estimates should not be based on effort or risk but on functionality. From this perspective, the smallest reasonable story is one whose implementation will enable one functionality.

This looks somewhat circular, as we have now moved from one fuzzy term (size) to another (functionality), in particular as in todays complex systems “one” functionality could be composed of a lot of different pieces playing together nicely. But requirement work is all about defining functionality in terms of specific, detailed user activities or system behaviors and as such is arguably easier to grasp and figure out than some number of story points.

It is usually the best idea to look at the functionality from the user’s point of view, because this will most closely resemble the business requirement anyway. But note that, without going further into architecture considerations, you might want to define functionality per system / component if you can define and observe the expected behavior per component. The “observe” part here is important as it is implemented, observable behavior that you can give feedback on. For the very same reason of getting feedback from humans, defining functionality from the user perspective is more preferable.

Now that we have discussed what the smallest story in terms of functionality could be, this might not be the most reasonable choice in your context. As a made-up example, consider the feature of where you want to save some document in some system. An additional feature could be to save a document under a different name (“save as”). From the implementation point of view, the difference in functionalities might be so small that it’s very easy to implement both in one go, so the team might decide that splitting off the “save as” is not worth it.

Another example could be where one user visible functionality (e.g. sending an order) requires the interaction of two systems (e.g. sending the order from some web application to some ERP system). Considering just one system part (e.g. the web application sending) as a user story might not be reasonable — it cannot be tested alone, hence doesn’t allow for feedback and also doesn’t provide any business value without the other system part. I believe this is where the entire team needs to give its input and consider not only the system functionalities but also (all) the tasks and the what is the minimum that makes a product increment valuable.

So now we have two different boundaries: whereas size in terms of functionality provides us with a lower boundary for a requirement (“implement at least one user-visible functionality”), size in terms of development effort provides an upper boundary (“can be implemented by the team in at most one week of an iteration”).

Clarity is required

Now, I believe there is a third “size” of a requirement which is related to clarity or level of understanding. Recall that at the start of many projects (or product phases) you’ll have a lot of very tiny and detailed requirements and some “bigger” requirements which have not been explored (or engineered) yet (which is what I would call “feature ideas”). An example of the former would be a user story along the lines of “As an editor, I need a save-as button right next to the save button so that I can give a new name under which to save my document”, whereas the latter would be something along the lines of “As a user, I want to save my preferences”. Quite obviously, the former is quite “small” and “clear”: it already says that a button is needed and what the expected behavior is. “User preferences” instead is just a label which might contain a huge list of things that need to be made customizable by the user and unless we break down this list into details, this requirement needs to be considered “big”. Another way of looking at this “big” story is that it is certainly not “ready” for implementation, so it is surely not a reasonable small-enough story. But even for the much clearer “save-as” story, there might be some lack of clarity on how to give the filename or other aspects (e.g. technical issues). Again, obviously we have a continuum between very clear and completely unclear.

Unfortunately, clarity alone will not be enough to decide whether a requirement is “small” enough: you can get very clear requirements that consist of lots of functionalities and are way too big to be implemented within one iteration (never mind huge use case or detailed specifications). However, from the perspective of “can we start implementing this requirement”, clarity of what needs to be implemented is the key ingredient that distinguishes a requirement from being “ready” from others. Clarity is a prerequisite to breaking it down into different functionalities and come up with a reasonable estimate (although it’s often also true that trying to break a big unclear requirement into smaller bits and pieces to come up with good estimates will help getting more clarity). Turning this around, even if you have a story that is reasonably small in terms of effort, if the team is raising concerns about the certainty of the estimates, the team is probably not sure whether it understands the functionalities that needs to be implemented and hence the story is probably still not “ready for implementation”.

Summing up, this leaves us with three important aspects influencing the “right” size of a story: clarity, functionality and effort. In the next article, I’ll explore some rules of thumb to split a big requirement into smaller parts.


Fun with function signatures

In a blog post on dependency inversion in Clojure I’ve discussed what this DI principle actually is about and the solutions Clojure offers to support it. There is one aspect that bugged me a little: For me, a fundamental challenge with DI in a language like Clojure is that you often have simple functions depending on simple other functions (simple here in contrast to protocols). In the article linked to above I discussed one way of resolving function dependencies by using an indirection over a service locator. However, in practice writing service locator lookups by hand was getting tiresome soon. So instead I decided to have some fun throwing together some macros that handling function signatures, which resulted in a small library.

Enter funsig

funsig shoots lower than Clojure protocols: it provides dependency management on a per-function level. What this means is simply that you can define a function signature with defsig and then provide implementations with defimpl. Implementations will depend on the signature. Let’s say we have some application code that depends on a printer function:

(ns my.onion)

(defn printer [string]
    (println string))

(defn print-account-multiplied [account multiplier]
    (let [result (* account multiplier)]
        (printer result)))

One might want more flexibility on how and where to print. In other words, one might want the application code (print-account-multiplied) to depend on an abstraction (printer) only and not on the concrete implementation as in this example.

Funsig allows you to inverse the dependency of the printer implementation. You would define the signature with defsig and have the appplication code depend on the signature like this:

(ns my.onion
    (:require [de.find-method.funsig :as di :refer [defsig defimpl]]))

(defsig printer [string])

(defn print-account-multiplied [account multiplier]
    (let [result (* account multiplier)]
        (printer result)))

You can then provide the implementation with defimpl:

(ns my.onion.simle-printer
    (:require [de.find.method.funsig :as di :refer [defimpl]]
              [my.onion :as mo :refer [printer]]))

(defimpl printer [string]
    (println string))

Note that the implementation has a dependency on the signature, not the other way around. Also, application code (print-account-multiplied) simply depends on the signature — here the signature is in the same file, but reference to the var in another namespace (i.e. using require\:refer) also works normally. For application code, this looks like dependency injection.

funsig will also allow you to have multiple implementations and then select the one you want.

Have fun!

You’ll find all the gory details in the README and / or in the intro document. The latter also explains the relationship to the service locator pattern mentioned above.

Feedback welcome!

Categories: Lisp
Defined tags for this entry:

Page 1 of 5, totaling 50 entries