Conway's law and the role of managers

Allen Holub had an interesting (post, tweet) toot about managers deciding team structure and software architecture:

Comic of a developer thinking about managers deciding the architecture. (c) by Luxshan Ratnaravi & Mikkel Noe-Nygaar under CC BY-ND 4.0
Conway’s Law by Comic Agile, CC BY-ND 4.0

If management is setting up team structure, they are designing your architecture. They know nothing about architecture, though. Self-organizing teams are essential. (See Conway’s Law). — Allen Holub

I mean, who am I to disagree with Allen Holub, but I agree completely with his conclusion but not with this argument. It might seem a little unfair of me to react to his tiny toot with a lengthy blog post, as I’m sure Allen could talk at length about the topic with tons of good insights, so please don’t take it as a criticsm of his post but as a way to express my thinking about how I look at Conway’s law and the role of managers.

Out of the ivory tower

First of all, I guess it depends on who you are referring to with “management”. If Allen is talking about top level management, then I agree that they usually don’t know about your architecture. But is this also true for the middle managers close to the teams doing the work? This level of management should know about architecture in my humble opinion, to avoid making foolish mistakes and to understand what the developers are talking about.

I think managers should today know about both architecture and Conway’s law. Ideally, they should also know about the Inverse Conway Maneuver (as eg. discussed in Martin Fowler’s article on Conway’s Law) which is simply the idea that you could structure your teams according to the wanted design of your system. Assuming the managers understand the architecture of the system (both in terms of how it’s structured currently but also a future target state), I think that managers can make use of this strategy when organizing their teams.

Communication is key

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. — Melvin E. Conway

Second, I think it is worth pointing out that Conway’s Law is not about how system architecture and team structure interact, but about the influence of an organization’s communication structure. Now, organizations are often re-structured with the implicit assumption that org structure matches communication structure. But this might be a bit naive, because organizations not only have a formal organization but also an informal one (see this overview of formal vs. informal organization). The latter is quite important for how communication will actually happen in my experience. So a change in the formal organization does not necessarily mean the relevant communication channels that drove your architecture before will suddenly disappear. In other words, people will remember that Eva knows a lot about the data access layer and will include her in relevant discussions, regardless of which team she’s in.

headquarters of the Romanian Architects Association
The headquarters of the Romanian Architects Association, built on the ruins of the Direcţia V Securitate, by Dimitris Kamaras, CC BY 2.0

Also, as discussed in Fowler’s article on Conway’s Law, it is questionable if Conway’s Law applies to all organization sizes. If you have two very small teams, in which communication among all people is easy and natural, there are a lot of other options than ending up with two components. Sure, if the communication between your people is complicated, you can expect to see this as friction in the architecture. So, if Fred doesn’t like working with Eva, he might be inclined to put in a completely new different data access layer. However, this might also happen if Fred and Eva are within the same group (I wouldn’t call them and their peers a team if that happens).

The Wikipedia page on Conway’s Law discusses supporting evidence that modularity of systems is directly related to how tightly coupled your organization is. So, managers shouldn’t just work on reporting lines but also on enabling good communication between the teams — while you might not be able to influence the informal structure, you can nevertheless try to set up good channels within the formal org structure. I certainly hope that you’re working in an organization which actively promotes cross-team collaboration and understands the benefit of networking, and not one in which you have to move up the org chart to find the highest shared person to convey information and to drive decisions.

End to end work

Third, managers should be aware of the idea that working in feature teams is usually better than working in component teams (cf. Roman Pichler’s more nuanced take on feature vs. component teams): the idea here is that a feature team would have all the capabilities to deliver a feature end to end, i.e. it does not depend on other teams. If your system already has a good design to support your feature development, then your cross-capability feature teams should be able to use and extend it. Naturally, if your system is big enough there might be more to know about the architecture then you can fit into the heads of your team — that’s when you’ll need again good communication channels across your organization. This will not invalidate Conway’s law but might enable that a change in the architecture isn’t just driven by one’s team decision alone.

Constant change is inevitable

Fourth, team and communication structure is a much richer and deeper topic that people have been thinking and writing about like forever and only alluding to Conway’s law ignores many other insights. E.g., team topologies is an approach that approaches an organization structure from the point of view of communication meachanisms and minimizing cognitive load of the people working within it. If you take a look at the team topologies in a nutshell you’ll see that it tries to also tackle the “pushing against Conway’s law”. If you follow the route as outlined in how to get started with team topologies you end up looking at the same fundamental pieces: the architecture and communication routes. That’s a tool that managers can use (and there are more, like Jurgen Appelo’s unfix framework).

And then you have all the good insights from Heidi Helfand that these days we’re constantly changing our team structures anyways, cf. Dynamic Reteaming. As she discusses, our teams change for so many reasons, people leaving, people joining, specific organizational needs etc. It’s only natural that a team structure will evolve over time, just like system architecture does (team topologies also expects to evolve your organization over time). It sounds a little bit over the top to me, to always pull the teams together and trigger a self-selection event in these cases (not that Alan is suggesting this). If we would reduce our thinking about team structure and reteaming to the technical or system architecture view only, we would swap one mistake (ignoring the technical implications) with another (ignoring people interactions and other needs), which would arguably be even worse.

Conclusion

So, what does leave us with? That managers should move around people like chess pieces, like they’ve always done and that’s that? No, of course not. As I said, I’m very much in agreement that self-organizing teams are essential. The people which are close to the work should have an important say in discussing the organizational structure. They can bring in the knowledge about architecture and technical dependencies, they will also bring in their preferences and personal needs and all of this is very important. But I also believe managers need to be involved, too, to clarify other needs to be considered (eg. business needs, future organization plans like hiring, etc). Plus you don’t want to burden engineers with having to handle the constant re-teaming all the time, because engineers usually don’t like it — otherwise they would have moved into management already.

Oh, and finally there’s a nice slide deck from Eberhard Wolff why he’s tired of Conway’s law which you should really click through if you’ve read this far.


Make it so! Decision making in software architecture

I find it interesting in discussions with developers that many have a bad picture of the job of a software architect. When thinking back to my first encounters with a software architect, who was performing his document reviewing job in a dedicated architecture department, I can totally relate, but then again it is 2021 and things should have changed [1]. After all, most developers have a very keen interest that the architecture of the system they are working on is good, but still some developers do not seem to be aware how much their work is impacting the architecture (and vice versa). “Every interesting software-intensive system has an architecture. While some of these architectures are intentional, most appear to be accidental”, as stated in Grady Booch’s article on “Accidental Architecture”. Some people now seem to believe that accidental architecture is the new norm, as “the best architecture, requirements and design emerge from self-organizing teams” (from the principle in the Agile Manifesto). In contrast, in organizations which have dedicated software architects, some people seem to think that it is the architect’s responsibility to ensure that there is a well informed design and to make the hard decisions as necessary. These hard decisions would probably be about “the important stuff (whatever that is)” that is called out in Martin Fowler’s article “Who needs an architect”.

But what does that mean for the daily work, should developers send all questions to the architect for her to make decisions and should architects sit in their ivory tower pondering them and ordering decisions by importance? Not in my view of the world. I’m a big fan of the role model that Fowler describes as Architectus Oryzus or acts as primus inter pares in Georg Hohpe’s article “Agile and Architecture: Friend, not Foe” in which the architect is closely working with the developers and who tries to identify important topics and make sure they are getting addressed at the right time.

Sketch image showing "A Design thinking Workshop", by Jose Berengueres, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons.
A Design thinking Workshop
Image by Jose Berengueres, CC BY-SA 4.0, via Wikimedia Commons.

Stefan Toth has a fantastic (German only) book on agile processes for software archictecture Vorgehensmuster für Software Architektur which describes patterns how to make this happen in daily practice. The core point of the book is that architecture should be a shared responsibility. Everybody should try to raise and address important concerns — whether there is a dedicated architect role or not is a separate issue. Toth lists patterns like identifying quality scenarios, listing architecture topics in the backlog, ad-hoc architecture meetings, common decision making and testing for architectural concerns (and many more) many of which are very team centric. The book is so great because it shows that it is easy to make architecture work a natural aspect of the development workflow and not some high ceremony.

In my experience, technical discussions are often concerned with architectural topics, so you might not even need to bring people together for formal architecture meetings — nevertheless technical design review meetings can be beneficial if they involve people that can and do share new perspectives but who would otherwise not be working so closely with the team. Similarly, if your team already adds technical debt and other non-functional items into the backlog, you probably are already following two of the recommended practices.

The role of the architect is then not so much the one of a decision maker but more the one of the person taking responsibility that architecture aspects are considered by the team. In bigger organizations this implies that the architect should be working closely with the development team but not only. While ideally the developers are also closely collaborating with any other important stakeholder (product owner, devops / operations, other dev teams), it is quite vital for an architect to understand the different (and potentially conflicting) quality needs of these parties. Again, probably even better than having e.g. 1:1 meetings between architect and security officer, so that the architect can point out operational constraints to the development team, would be if the architect tries to make sure that a security expert is involved in design and requirement discussions.

Hippo with tongue stuck out (via Pixabay)
Highest paid person’s opinion.
Image taken from Pixabay

However, the job of an architect is not only consisting of organizing meetings with the right people. In the end, intentional architecture is in need of decisions. “Common decision making”, as listed above, means two things to me: first, people who are affected by a decision (e.g. the developer who has to implement the result) should be directly involved in the decision making process. In teams that do not have a dedicated architect this seems naturally to be the case, but it is worth pointing out that this can easily degrade to single programmer decisions. The other anti-pattern here would be that the architect, likely to be the highest paid person in the room, makes a HiPPO decision or specifies a design while at the same time not being close enough to the problem and / or the existing code base. [2] The effect here can be devastating if the decision turns out to be wrong: not only will people think that the architect did not listen to the team but also that the architect might not have the necessary competence. This can result in serious psychological safety and trust issues. A good working approach to avoid such problems is running small experiments (spikes) to validate options.

The second aspect is that a decision is made in the first place, which is even more important. The idea of decision in the last responsible moment notwithstanding, I find it surprising how hard it can be for organizations / a group of people to take decisions. It is important for an architect to be able to guide a group towards making a decision but also to be comfortable to make a decision themself, if the group cannot find an agreement. The biggest issue here are the trade-offs of potential solutions: e.g., one solution might have benefit A (e.g. scalability) while being weaker in B (e.g. ease of use) and another solution has opposite characteristics. Often different people will have different opinions whether A or B is more important in the selection process. It is very helpful for teams to come to an understanding (not agreement) about these different weighs that people give to the characteristics and then hopefully, the architect can explain why she thinks that one argument is more important than the other. Coming to a decision is more important than convincing everybody, though: aim for solutions that everybody in the team can agree with, even if they would still prefer a different one. Also, it is a good idea to document decisions in a light weight fashion (e.g. in a small Architecture Decision Record).

In case that during implementation the developers run into obstacles, it is quite likely that the developers will actually figure out a different solution than the one decided. This is another case where the “architect as lonely decision maker” can lead to a lot of unnecessary friction. In practice, the choice of a different approach is not a problem as long as it is handled in a similar responsible way, where the developers take a look at the implications and trade-offs of their choice and involve others (and, please, update that ADR).

Footnotes:

[1] I am obviously not talking about so-called Enterprise Architects who are mostly concerned with governing how many systems at a company interact to provide the most business value. Instead the focus here is on software architecture for a single system (which, of course, is likely to interact with many others).

[2] Sometimes the opposite scenario might be true: the team is consisting mostly of people which do not have enough expertise, whereas the architect has long experience with the code base. In this situation, the architect can be helping the team by sketching a design approach and sharing his knowledge. Still, there should be room for questions and further ideas from the team.


Stand and deliver -- how can a team become quicker?

Recently I met with an old colleague of mine and we talked about teams, among other things. He asked me what I would do to urge on a team to become quicker and to deliver more. My gut reaction was that this is a thin line to walk. The following delves a bit deeper into the issue.

What not to do

One particular pitfall is to ask the team to deliver more story points. I guess the motivation behind the idea is two-fold:

  1. Delivered Story Points are easy to measure.
  2. The team could work harder and then easily deliver more.

One of those assumptions is generally speaking false and it’s not the measuring point. What I usually see is not people slacking off but people struggling.

But it’s not only that this core assumption is wrong, it gets worse when you consider the likely effects of asking for more story points to deliver. First of all, note that it’s very easy for the team to deliver more story points: The team can simply raise their estimates for each story. They can raise the estimates basically on each estimation, thereby gaming the system and destroying any trust you can have in the estimates. This essentially means you’re loosing the velocity as an indicator what the team can reliably deliver in the future.

The second effect is even worse: If people feel pressure to move more stories to “done”, people will concentrate on delivering the core value — this being the functionality that the client / product owner was asking for. On the flip side, they will stop caring about quality because that is apparently not what is the main driver. As a result, people might stop doing the following things, because they actually cost time, time they can’t spend on finishing the next story:

  • testing thoroughly
  • covering the changes with automated tests
  • cleaning up the legacy code
  • refactoring your first “working” implementation into a maintainable shape

The biggest problem with this is that the effects of all this will usually not affect the current sprint, but will probably cause severe slow down later on, either because of bugs that need to be fixed later or because all the technical debt will make any further changes so much more complicated. Let me repeat this: if you don’t care deeply about the quality of what you build, the effects will slow you down, maybe not today, but certainly tomorrow.

Don’t ask for more points, instead drill deeper to figure out what is holding back the team and focus on fixing that.

Why the team can’t move faster and what to do about it

My friend then came up with the idea that it might be viable to look beyond the usual Scrum process, maybe consider Kanban or something else. My perspective (and reaction to him) is that he was on the wrong track. Yes, Scrum does add some rituals that some people consider to be unnecessary baggage that don’t add value. In my experience that’s either because the team in question is actually already a highly experienced team that delivers very successfully or because the process parts are executed so poorly that they don’t help the team. But usually the problems are elsewhere and simply changing the framework you’re using will not fix those.

Process problems

Let’s be clear: yes, in some cases you actually have a problem in the way work flows through your system. One classical scenario is that all testing work is only started at the end of a sprint, when all the implementation is done or you have too many people working on items in parallel, not being able to finish anything early. You are basically setting yourself up to find bugs too late to be fixed and very often also to have left-over testing-only work that needs to be completed in the next sprint.

If you’re using any method to inspect and adapt (e.g. regular retrospectives), this is one of the things you would expect the team to be able to identify themselves and also to take action on. Some guidance from an experienced Scrum Master can be really valuable here, she could for instance use an activity to identify the value stream through the team. Also, ideas from various frameworks can help, e.g. setting a work-in-progress limit as done in Kanban.

The other classical case of process problems is where the organization is putting up several hurdles in front of the development teams without even recognizing it. This could range from forced use of ineffective tools or low-level machines over tons of bureaucracy to using outdated micro-management practices. These process problems are far from easy to tackle and require usually a lot of help from somebody outside of the development team (e.g. the Scrum Master or some manager).

So, yes, sometimes your process is a problem, but it’s unlikely that it’s a question of Scrum or Kanban (or something else).

Worthless work doesn’t motivate

If your team doesn’t seem to have high spirit to tackle the next job, it’s probably not related to your organization. If your product is really worthless, no one will be motivated to put in any effort. I have a good friend who was literally asked to add piles of sh*t to a game his employer produced. This was a rather short lasting work relationship. I guess such an experience is the exception but nevertheless in some cases people see no value in what they produce.

Let’s assume you team is working on a genuinely useful product. This does not mean that it’s obvious to everybody that the next version or feature serves a useful purpose, too. Sometimes the people which know best why delivering a product to the customer is important are the same which don’t understand the need to explain this to the development team. But this is vital for motivating the good people that are building the product in the first place. Ignoring this often goes along with a mindset of seeing the development team as a feature factory, a black box into which you have to feed specifications and which will spit out the next version of the product on the other side.

The problem is that on the inside of the factory the value of the work remains in the dark, of course. Either feature that needs to be built is just another feature, all different but also all the same. As a result, team members will get the impression that it’s actually not all that important what exactly they are building, outside of meeting the deadline, of course. Why there is a deadline in the first place and what happens if it’s missed or whether the actual customer problem could be solved better with an entirely different solution, is a problem of other people “above my pay grade”. Of course, if the team doesn’t understand the value of what they’re building, how could they care about delivering something with better quality or with higher efficiency?

I’m convinced that the root cause of this issue is that there is no collaboration between the product managers and the development team. Try to live up to the idea that “Business people and developers must work together daily throughout the project”. It’s hard to just throw requirements over the fence without any further explanations, if the guys can ask all sorts of questions every day in the stand-up. Day to day collaboration and just spending a lot of time together (aka “socializing”) will also help with the insight that the developer team consists just of fellow humans with a brain and a genuine interest that their work should matter.

The other answer to this wide-spread problem is to make sure that at the start of a project (release, product increment, sprint etc.) the person responsible for the success and the money explains the vision and the goal that needs to be met. If you can back up this story with customer input or other market research, the better. When explaining and discussing things (the what), keep referring back to the goal and the vision (the why). This will lead to better discussions, better solutions and almost certainly to higher motivation among the team.

Technical debt

“If your code is crap, stickies on the wall won’t help” — Rich Rogers

Stickies on the wall The truth of this quote can’t be overstated in my experience. The more technical debt you accumulate, the slower your team will become as it struggles to understand all that crap. Also, code that is hard to understand will make it easy to introduce new bugs and each new bug means that your team will not work on new features. Bad code will also demotivate your people as they feel like Don Quixote fighting an endless battle against windmills that will constantly throw more bad code into their faces. There is also the broken-window syndrome: if people get the feeling that everything looks awful and is broken already, they can get the feeling that it is alright to put in even more spaghetti code.

“Continuous attention to technical excellence and good design enhances agility” — Agile manifesto

I think it’s a shame that the focus of agile has shifted from methods like XP to frameworks like Scrum or SAFe which don’t have the same emphasis on technical excellence. Don’t get me wrong, I know that a lot of the techniques which were promoted in XP like TDD are now considered state of the art for software development, regardless of the process for work organization you are using. But the thing is that it’s too easy (for people like the Scrum Master) to concentrate on getting stuff “done” and other core pieces of such frameworks and to let the technical excellence slip completely out of focus.

The right action to take is to make people constantly aware that good technical solutions which are maintainable will make life (or at least work) so much easier and nicer. Use a Definition of Done which is focusing on quality and maintainable code. Establish pair programming and code reviews. Make sure you get technical debt items into the backlog and that they are also prioritized appropriately.

Note that this is not about “gold plating” and spending endless hours on polishing code. But honor the Boy Scout rule that you should leave any place you visit a little bit cleaner than before.

Lack of expertise

One related problem is that you might have people in the team which don’t have all the necessary expertise that would allow them to work “faster” or to find high quality solutions. This usually comes in two flavors:

  • lack of general development know-how (e.g. SQL, programming language, design patterns etc.)
  • understanding of the existing code basis

Both are usually problems which will become smaller over time because your beginners will gain more experience in the fundamental technology and also in the code basis on which they are working. Be aware, however, that it’s easy to reach a level where people have learned enough to be dangerous. That’s difficult because they can both produce something that “works” but is not of the required quality. It’s easy to stay on this low-level plateau of understanding when there is no mentoring / coaching beyond this level.

Clueless Cartoon Man Looking At Different TimesThe key idea to solve this problem is to actively invest into the expertise of your team members. One idea is to have dedicated training sessions addressing the required know-how areas. However, getting a theoretical introduction into a topic is usually not enough and should only be the starting point to get hands-on experience. So, I think it is crucial that the overall team understands that the more experienced people can fundamentally help the less experienced people while doing the real work — by patiently answering questions, explaining concepts and reasons, by pair programming and general technical coaching. Note that this isn’t a one-way road (from senior to junior) either: this becomes a lot easier if the inexperienced people are confident that it’s okay to ask for help and to admit failures without having to fear retaliation — which brings us to the next point.

Team conflicts

The agile manifesto prefers “Individuals and interactions over processes and tools”, but it’s so much more easy to focus only on the latter. Especially technical people are often too happy to ignore people problems (maybe that’s why they chose to go into tech in the first place). Team conflicts can come in various different flavors, e.g. some people might not want to work together, maybe there are blame games being played or there are bad rumors being spread behind people’s back. You don’t have to be able to recite the proverbial Five dysfunctions of a team to understand why deep conflicts within a team will be a fundamental blocker on the way to more effective and efficient results. Emotions will get in the way of any discussion, technical or other.

While it is maybe obvious to have this insight into the damaging effects, actually figuring out that there is such a conflict in a team might not be so easy if there is no open fight. Fighting openly is very often avoided by all involved parties, in the name of so-called professionalism. Even if it becomes clear that there is a conflict (or multiple), the team might be very reluctant to actually discuss or resolve it. This is directly related to the conflict, of course, because people need at least some feeling of psychological safety (and ideally also trust) to be able to address problems openly.

I think it’s rare that deep conflicts within a team are resolved without involvement from the outside, regardless of whether the external person is a manager or just another colleague. This takes a lot of patience and even more empathy. An important ingredient here is to make sure that the emotional aspect of the conflict are actually coming to light.

Ironically, the prime directive for retrospectives that is intended to remind people to be respectful can sometimes be a hindrance to openly discuss the conflict. People can misunderstand “believe everybody did the best job they could” that no criticism should be voiced at all, where it’s supposed to enable trust to speak up openly about problems. Activities like Spot the elephant can help here. In my opinion, it is okay when this results in hard discussions and hurt feelings and when emotions become apparent. However, it is the responsibility of the moderator to ensure that these emotions don’t get in the way of a discussion of solutions. Ideally, the involved people should be the ones which come up with the solution. Unfortunately, in the case of deeper personal conflicts, often the involved people have no idea how to separate factual discussions from emotional aspects or have no experience how to give or react to feedback without deepening the conflict. Moderation from some experienced “outsider” is key here. Of course, in some cases, the best resolution might be to change the setup of the team, but this amounts to losing the game.

Keep it simple, stupid

In my experience, the main problem in software development is rarely the technology or the main process, but almost always people and their interactions. Note that also technical debt is coming either from inexperience or from people not collaborating enough to point out the rising quality problems. So, if you think the process is the problem, I would recommend to take a deeper look.

ObTitle: Adam & the Ants, “Stand and Deliver” from “Prince Charming”


Technical debt equals missed quality requirements

Technical debt assumes that you have an existing system and you know already about the areas where the duct tape is becoming thin. Let’s discuss the problem, the conflict around technical debt a little. It’s basically always the same battle: the guys maintaining the system probably have good reasons why they want to pay back on the accumulated technical debt, whereas the product owner believes more functionality is more important.

This is a typical clash of interests between people from different tribes and usually from different departments: the developers report to a technical manager, the product owner to a business guy. The IT manager is typically under pressure to minimize costs, in particular costs for support and maintenance, so he’s interested in paying back on technical debt during projects. The business manager, however, is under pressure to convince new clients with new features. So, that there is a difference in how technical and non-technical people set priorities between technical debt and new features doesn’t come as a surprise: naturally, to the business side this is all about technical issues which they don’t consider to be their concern. It’s only natural that they believe that tackling such technical issues should be solved by the technical folks “somehow”. After all, it were these guys which build the system with all these problems in the first place, and now they want to spend more time and money on this? Although I’ve exaggerated here a little maybe and this line of argument is way too simplistic (given that often debt is accumulated over time), it’s still quite popular. Such product owners seem to believe that they are only responsible for developing new functionality. Corporate culture can contribute to this: some companies have a culture where every new glitter is taken for a star and gets way more attention than the cash cow functionality that keeps existing clients happy.

So some product owners will see technical debt as a separate issue which needs to go on a maintenance budget, a budget that somebody else is responsible for, e.g. the IT department. The problem with that idea is that in a culture where building new features is the only thing that counts, typically no one ever gets around to clean up the technical debt. All what will happen is that you’ll have some people that fix bugs with this maintenance budget. So you don’t actually ever spend the money on technical debt but only on the results of it which of course doesn’t address the root of the technical issues. Also, if you finally do get an approval for a refactoring project, these are really the most boring and horrible types of projects to work on. So it’s not likely to see a lot of happy, motivated people on such projects — and sure enough the most competent people are probably assigned to more valuable projects anyway. As a result and because there is no business pressure on such refactoring projects, as soon as something else comes up, such refactoring initiatives are abandoned or at least down-sized to a minimum. Peter Seibel describes in a recent, lovely written article the need for an Engineering Effectiveness team at Twitter in which he lays out in some details how hard it is to really put up with technical debt.

Let’s drill down on technical debt from the point of view of requirements. Wait a minute, this doesn’t sound right — how could technical debt be a requirement? Obviously, it isn’t, but there is a close relation. Technical debt can come in different flavors: e.g. you really should install a second machine and a load balancer so you’re prepared for failure or we really need to rewrite module foobar.clx to finally get rid of all the spaghetti code that’s slowing us down to oh, we really need to support responsive design, we’re getting more mobile users every day. Now, if you take a look at these three simple examples, they are related to quality requirements: the first is about availability, the second about maintainability and the last about usability. Take a look at the ISO 25000 standard for software product quality page listing all the quality attributes one might want to consider. Technical debt is always tied to some quality that your system should offer, but doesn’t.

A reason for this is that quality attributes are often not made explicit during requirements gathering, regardless of whether you’re following an agile approach or not. The less technical the product owner, the more often they seem to assume that performance, scalability, security and the other -ilities will come out just right by magic. Technicians, on the other hand, know perfectly well that they will not. It’s worse if they don’t that: they will build something that might fulfill the functional requirements but not the non-functional requirements. “Works on my machine” might be fine for a naive developer, but nobody will be happy if takes 5 minutes to load the page for the gazillions of parallel requests in production. There is a reason why software architecture is mostly concerned with quality, if you don’t plan and build for scalability, it’s unlikely you end up with a highly performing system. Take a look at the picture: this bridge wasn’t planned to cope with the amount of traffic it sees nowadays, some time in the 80s somebody decided it would be okay to have three lanes instead of two and didn’t think through the long term consequences.

Rheinbrücke A40 Duisburg-Neuenkamp
The bridge of the highway A40 over the Rhein in Duisburg. It is damaged so bad that trucks are no longer allowed to cross it.
Picture made by kaʁstn Disk/Catstitched by Daniel Schwen, licensed under CC BY-SA 3.0 de over [Wikimedia Commons]( https://commons.wikimedia.org/wiki/File:Rheinbruecke_Neuenkamp_pano.jpg#/media/File:Rheinbruecke_Neuenkamp_pano.jpg)

Of course, technical debt can also accumulate over time, one might rightly point out: sure, one machine was enough for the requirements at the start, but nobody followed up on the increase in users and nobody cared (or had time) to clean up the code in module foobar.clx back then (nor in the following six years of quick bug fixes and minor one-off patches that have grown on it like leeches). This is a sign that nobody actively had an eye on how the world in and around the systems changes and to take action early on. For code quality, Uncle Bob points to the Boy Scout rule which says that you should always clean up (regardless of who messed it up) — another way of trying to ease maintainability by paying back on technical debt every day.

The over-aching point here is that you, as a technician and you, too, as a product owner need to think about the quality requirements of your system and to do that over the entire life cycle. What was a fitting solution at a time might turn into technical debt over time. This means means your system now doesn’t hold up to the quality requirements you and your clients needs and no matter what was the cause for it, you’re better of fixing it now than accumulate even more technical debt.

Isaac Sacolick describes in his article on How to get an agile product owner to pay for technical debt how to address the common problem of technical debt heads on, mostly from a managers perspective via process and by making people (mostly the technical lead) responsible. However, I think it makes a lot more sense to try to ensure a common understanding between the developers and the product owner. One way to go about this is to use quality scenarios: as a member of the dev team, ask your product owner how many users the system needs to handle. Ask her if she thinks it’s okay if somebody might find a security issue earlier than you. Or if she thinks it’s okay when a seemingly small change will take three weeks because the code is an intangible mess that nobody understands since Dieter left? These sort of questions hopefully open up to discussions based on business value, ROI and cost of delay. You are then using the language product owners understand and you might also learn something along the line (no we expect so many users to do X in the future, so investing in module Z doesn’t make sense).

Now, granted that doesn’t always work. An old colleague of mine always denounced the old system he worked on as being exactly the way it was ordered. Isaac’s article might offer some advice here, of which the most important one is probably that paying back technical debt is way easier if you have a CIO or someone with similar power supporting you.


Dependency inversion in Clojure

The problem

I was recently reading a nice German book on Effective Software Archictecture by Gernot Starke and stumbled upon a discussion of the dependency inversion principle, which got me thinking. Gernot Starke first discusses the problem with an allusion to traditional procedural programming (translation mine):

Classical designs for procedural languages show a very characteristic structure of dependencies. As (the following figure) illustrates, these dependencies typically start from a central point, for instance the main program or the central module of an application. At this high level, these modules typically implement abstract processes. However, they are directly depending on the implementation of the concrete units, i.e. the very functions. These dependencies are causing big problems in practice. They inhibit changing the implementation of concrete functions without causing impacts on the overall system.

Classical dependencies in procedural systems He then goes on to introduce the idea of programming against abstractions and introduces the idea of the dependency inversion principle, first coined in Bob Martin’s DIP article (see also another thorough discussion in Brett Schucherts article on DIP). Basically, the idea is that the integrating process refers only to abstractions (i.e. interfaces) which are then implemented by concrete elements (classes), cf. the next figure.

Integrate with abstractions When I take a look at some of my recent Clojure code or at some older code I’ve written in Common Lisp, I immediately recognize dependencies that correspond to those in a classical procedural system. Let’s go for an example and take a look at one specific function in kata 4, data munging:

(ns kata4-data-munging.core
    (:require [kata4-data-munging.parse :refer [parse-day]]
              [clojure.java.io :as 'io]))

(defn find-lowest-temperature
    "Return day in weatherfile with the smallest temperature spread"
    [weatherfile]
    (with-open [rdr (io/reader weatherfile)]
         (loop [lines (line-seq rdr) minday 0 minspread 0]
        (if (empty? lines)
            minday
            (let [{mnt :MnT mxt :MxT curday :day} (parse-day (first lines)) ;<-- dependency!
              curspread (when (and mnt mxt) (- mxt mnt))]
            (if (and curday curspread
                  (or (= minspread 0)
                  (< curspread minspread)))
               (recur (next lines) curday curspread)
               (recur (next lines) minday minspread)))))))

The dependency here is on the concrete implementation of parse-day, you can basically ignore the rest for the argument here. Given that this was a small coding kata, this is not unreasonable (and in the course of the kata, the code changes to be more general), but the issues here are obvious:

  • if we would like to parse a weather-file with a different structure, we have to change find-lowest-temperature to call out to a different function,
  • if the result of the new function differs, again we have to change the implementation of find-lowest-temperature,
  • we also have to change the namespace declaration, i.e. we probably want to require a different module.

Clojure’s built-in solutions

The application of the dependency inversion principle is typically shown in the context of object-oriented programming languages, like Java where you use interfaces and classes implementing those interfaces for breaking the dependency on concrete implementations, cf. the figure above again. But as we’ll see the principle can be applied independently of object-orientation. I’ll discuss higher-order functions, protocols and multimethods as potential solutions.

Higher order functions

For starters and probably painfully obvious is to make use of the fact that Clojure treats functions as first-class objects and supports higher-order functions. This simply means that we can pass the parsing function as an argument to find-lowest-temperature.

(defn find-lowest-temperature
    "Return day in weatherfile with the smallest temperature spread"
    [weatherfile parsefn] ; <-- function as parameter
    (with-open [rdr (io/reader weatherfile)]
         (loop [lines (line-seq rdr) minday 0 minspread 0]
        (if (empty? lines)
            minday
            (let [{mnt :MnT mxt :MxT curday :day}  (parsefn (first lines))
              curspread (when (and mnt mxt) (- mxt mnt))]
            (if (and curday curspread
                  (or (= minspread 0)
                  (< curspread minspread)))
               (recur (next lines) curday curspread)
               (recur (next lines) minday minspread)))))))

This way, we can simply call (find-lowest-temperature "myweatherfile" parse-day) and freely substitute whatever file format and accompanying parse function we need. What does this buy us?

  • We no longer have to modify find-lowest-temperature when we want to use a different parse-day function.
  • The namespace containing find-lowest-temperature also no longer requires the (namespace containing the) parse function.

But there is also a down-side: find-lowest-temperature assumes that all parsing functions it will get fed adhere to a signature that is entirely implicit: parsefn needs to take exactly one line and needs to return a map with given key-names. Higher-order functions don’t provide a solution for this per-se, so in order to solve the implicit signature issue we need to look elsewhere. This is nothing Clojure specific: Assuming you’ve passed in an object either as a method parameter or via Setter-Methods or Constructor-Injection (cf. dependency injection), Python’s or Ruby’s duck-typing basically works the same way: the caller of a method simply assumes that the callee offers a method with the right signature. It is the responsibility of the caller (of find-lowest-temperature) to provide a matching function for parse-fn.

However, this actually amounts to just move the problem from one level to the next: now some other level has to decide which concrete parse function needs to be used. This next level will have again the exact same problems: it will depend on both concrete implementations of find-lowest-temperature and parse-day (or any other parse function). If you think this through, it’s obvious that in general at one point or another, you have code that determines which function to call and which parameters to use. The question is only if we can use abstractions or whether we have to use concrete implementations. We’ll return to this issue, that now at some other level you need to handle the problem, later.


Page 1 of 1, totaling 5 entries