Entries by Holger Schauer

In the last article on the right size of a user story, I discussed some aspects that contribute to finding an answer that works in your context: clarity, functionality and effort. In this post, I want to discuss the question on how you can move from a big feature to smaller stories. The assumption here is that we already understand the requirements “good enough” (i.e. maybe captured in a use case document), so we have reached already a quite clear picture of what needs to be done.

By Chmee2 (Own work) [GFDL, CC-BY-SA-3.0 or (CC BY-SA 2.5-2.0-1.0 ], via Wikimedia Commons

There is quite bit of literature on these questions, with the book “Agile software requirements” from Dean Leffingwell being one particular source of input (the SAFe page on stories lists an overview). Being asked, what would you recommend, the usual consultant answer would be “it depends”, but I think it is safe to come up with some rules of thumbs.

0. Don’t split

The bottom line for story splitting is that you should avoid it if you can. Having a lot of very small stories will come at a price — loss of overview is an important one, and if taken too far, it can also become hard to reason about the business value of a story. Formulating the “… in order to” part of the classical user story template (“As a <role>, I want <action> in order to reach <my goal>”) is often a good sanity check: if you have a hard time finding the wording for “my goal” is usual a sign that you’ve split up the requirement too much. There is one classical mistake that can easily result in this situation: if you break down a requirement by components or systems, it’s highly likely you will end up with problems to formulate a reasonable business goal — most probably you split off an implementation task, cf. below.

Assuming the main reason is to fit the work into one iteration (or one week of the iteration, cf. above), if your story is already “small enough” in terms of effort, then there is not much point in going for even smaller stories. You will run into the occasional discussion that a story should ideally be able to finish (“move to done”) within one day, but it should be up to the team to decide whether pushing for this really brings any benefits like additional helpful feedback. More often than not, the team might just try to game numbers, e.g. the number of stories they can finish. There is an obvious tension here between the benefits of small stories (delivering working code early, enabling quick feedback) and unnecessary overhead (splitting in a meaningful way, documenting, managing and tracking the bigger amount of stories), that needs to be balanced according to the situation at hand, but try hard to avoid splitting into too tiny details.

The other question is, do you need to split the requirement now? If the requirement does not need to be elaborated now, because it is not planned for the current or the next iteration, there is no need to split it now — potentially this requirement could change over time anyway, so preliminarily driving for details might actually cause more problems than help. For instance, when a bigger feature involves multiple workflow steps and you split the feature into stories, where each implements one step at a time, you might have a hard time in your backlog tying these pieces / steps together, if you want to revamp these steps in its entirety.

1. Do the minimal thing first

If you think you need to split down a requirement, try to identify the “minimal viable functionality” as the first story. This is the minimum functionality that the system needs to provide that brings some (potentially rather minimal) benefit to the user. This functionality becomes the first story, whereas all other aspects of the requirements are addressed (as extensions) in separate stories.

Which parts provide the minimal usable functionality?
Foto assumed to be by Pavel Krok~commonswiki under [CC BY-SA 2.5], via [Wikimedia Commons]

As an example, let’s again think about user preferences for some system: the minimum useful functionality for a user consists probably of setting the preferences, saving, loading and applying them. Simple counting these steps leads you to the observation that this alone consists of at least four different functionalities. However, exporting my preferences to a file and importing them on a separate system is certainly not “minimal”. Depending on the intended users, even providing an inline editor to set the preferences might not be part of the minimum functionality. These functionalities could become separate user stories. The drawback of using this approach is that the resulting stories are not truly independent any more: you cannot prioritize exporting and importing preferences to be done before setting, saving and loading them.

Now, of course, this minimal viable functionality might still consist of a lot of work which might still be “too big” (in implementation effort, not fitting into one iteration). Here all the ideas from Dean Leffingwell are applicable, in particular trying to split by persona or business rule variations, data entry methods or data variations or workflow steps. In our example, setting preferences and saving as well as loading and applying preferences could be two different workflow steps (and hence different stories). However, this approach would probably result in stories that are too small according to rule of thumb 0 above. When splitting by workflow steps, try to have a minimal complete workflow in one story or at least plan to have this minimal workflow implemented in one iteration.

2. Optimize later

A good guiding principle in programming is to avoid premature optimization. While the principle talks mostly about optimizing performance of computations, we can also use it as a rule of thumbs for splitting requirements. The idea is to to take the simplest approach that could possibly work. This has a rather broad applicability in terms of technical aspects (architecture), but also usability and other so-called quality or non-functional requirements. After implementing this first rough version, use the feedback to check if it is “good enough” — e.g., whether performance, usability or security requirements are met already. If not, optimize these “-ilities” later on.

Some people might find this suggestion offensive. Security, for instance, is one quality that many claim cannot be designed into a system as an afterthought. But this is also not what I’m suggesting. Instead, I’m suggesting to formulate these “-ilities” requirements separately to isolate the functional core of the big requirement and to implement the optimizations later. This is quite independent of the need to design them — whether you design them upfront (as this might be part of your constraints in a “Definiton of ready”) or within a Sprint is a different question, but of course you need to think about the required qualities that the functionality needs to have. The idea here is simple: If your story is too big, put the proverbial lipstick on the pig in the next round. If you are afraid that your product owner will not care about security and will lower the priority of implementing these security stories to the point that they will never be implemented, your product has a bigger problem anyway (and probably security problems already) or the usage scenario that the product owner has in mind might be different from your expectation. Probably in both cases spelling out the required quality (and the accompanying work and effort) in a separate story could even be helpful to spawn a discussion.

That being said, breaking down your stories by functionality should be preferred. This is especially true if your stakeholders have some expectation of these qualities — if you miss out on these qualities, you’ll likely receive feedback to that effect. At least, you’ll be able to point to another story then to address it. Again, finding a good balance is important, but also not always easy.

3. Avoid splitting by technical components

Who would prefer the naked pizza crust over a slice of the real thing?
By BrokenSphere (Own work) [GFDL or CC BY 3.0 ], via [Wikimedia Commons]

Technical user stories that are intended to cover work specific to some component are an anti-pattern. Typically they are brought up by people which are still stuck in the old trap to split work according to responsibility for features vs. components, cf. the discussion of features and components in SAFe. Don’t split by technical components unless the component can deliver business value / user functionality on its own.

Typically you cannot test some functionality in a reasonable way for some component on its own (e.g. the UI alone, not connected to the database). Hence, you will also not be able to get any feedback for the implementation of the story. E.g., saving and loading your preferences from the database is certainly a reasonable technical task, but how would your stakeholder test this without the UI being implemented? How would your product owner find out that there is a minor problem in the saved data if the only way to access it requires sending a web service request or proficiency in your programming language of choice? So, even though you might have finished the work on the component completely, you will not get the quick feedback that we would like to enable with small stories. And, of course, the component alone will also not be delivered as a result of your iteration, so it’s also not adding any business value at this point either.

This rule of thumb also can be debated. There are situations where one component can provide value already — e.g. a CMS system where you extend some functionality on the authoring environment first, so the editors can easily start using it. Even if the readers cannot yet see the result of the new features, some business value might already come from easing the work the editors. This case should not be confused with a different line of reasoning: For the UX designer, for instance, it might make a lot of sense to provide feedback on a UI preview that is not connected to the database. Your DB admin might already point out problems with the schema etc. However, this is inter-team feedback then, not customer feedback. In the end, the measure of progress in agile projects is delivering working software to the customer — and from a user perspective, both of these examples fall short, as essential functionality is missing. Instead, go back to rule 1 and try to identify the minimal complete slice of functionality that delivers value to the customer.

Task splitting and Spikes

There is, however, of course also a time when it is exactly the right thing to do to break down a story into technical tasks that need to be finished: this happens when the story is ready and due to be worked on given its priority, typically as part of a Sprint Planning 2 meeting in a Scrum setup. Especially for engineers it is easy to fall into the trap to confuse story splitting (as requirement work upfront) with task splitting (organization of concrete work within a sprint). A focus on technological considerations (and often also personal priorities) can quickly lead to splitting work such that it is developer-friendly, but does not deliver results to customers quickly. Let’s discuss some aspects of how task splitting influences work organization.

Stating the obvious, splitting technical work into tasks requires technical understanding, both of the system in which the requirement is implemented and of the technologies involved. But it is also not always the case that the team understands the system or the technology well enough. In this case, the team can decide to spend some time on research work to figure out what exactly needs to be done. Don’t define this as a “research implementation” task and plan to complete the entire story, you likely have no idea how much work it takes to complete everything anyway. Instead treat the research work separately, this is usually called a spike. As the goal of a spike is to learn something and not everything, a strict timebox should be defined (usually not exceeding one or two days), after which the spike is called of, regardless of how much insight was generated. Spikes usually are hence not estimated in story points, but reduce the capacity of the team, so you should also try to not have more than or two spikes per iteration.

Work in parallel to get it done
By OSU Special Collections & Archives, via [Wikimedia Commons}

Splitting a story into tasks follows different rules: for instance, splitting by components is usually a good idea, especially if your team does not consist only of full-stack developers. Also, even with full-stack developers, you need to start somewhere, which good software architecture practices would demand to be clarifying interfaces between the various components. Which brings us to one big issue of task splitting: Identifying dependencies between tasks. These are usually directly following from the organization of the components. Of course, as with user stories, you should strive to avoid dependencies in the first place (and use dependency injection and other mechanisms to decouple your dependencies). This is not only good engineering practice, but also allows to parallelize the resulting work (i.e. have multiple people work on the same story). Teams sometimes split their work such that only one developer is actively working on a story that consists of multiple tasks. This usually leads to the situation that many stories are started but they all take a long while to be completed. Instead try to distribute work between multiple people: the frontend ninja can start his JS foo, while the backend expert whacks the database in parallel, modifying the previously defined contract as required during the work. If the team can work on multiple tasks of a story in parallel, then this does not only encourage collaboration and team spirit, it will also ensure that the highest priority work items are finished first.

But the tasks to complete a story usually also involves other work besides coding, e.g. UX work or manual exploratory testing. More often than not, these are not happening as agile “on the spot” as other work, instead it’s a quite common anti-pattern to treat an iteration like a mini waterfall: UX (of everything) happens at the start, then development, testing (of everything) happens at the end. This can lead to a lot of unneccessary work done upfront (UX for stories which the team actually doesn’t get around to) or too late (testing starts so late it cannot be completed in the Sprint). Again, try to do the work only when it is necessary and try to do more stuff in parallel.

One of the recurrent puzzles with regard to requirements management that I keep running into is the question what would be the “right” size of a requirement. It often starts with some people coming up with very tiny and very specific requirements and then some very huge, completely fuzzy ones to go along with them. Then they have heared or read somewhere that in agile setups requirements should be small and wonder what this means and how to get there.

Advanced Scrum
Cartoon by Oliver Widder, licensed under CC BY 3.0 from [Geek&Poke]( http://geek-and-poke.com/geekandpoke/2016/11/6/advanced-scrum)

These huge requirements are often called “epics”: An epic is a requirement that is either felt to be too “large” or is too “fuzzy” and one way to avoid never-ending dramas is to split these into tinier user stories. Actually, here we already have two very distinct aspects: a requirement engineering question of how to move from unclear feature ideas to concrete detailed requirements on the one hand and a requirement management question on how to organize requirements such that they are appropriately prepared for implementation. I discussed the requirement engineering part in an article on agile conception already, so I’ll focus on the latter part here. This size question is actually two-fold: What is a reasonable small story? And how can we break down a big feature into several smaller stories? I’ll address the first one below, the next one in an upcoming article on splitting requirements.

Why small stories?

Let’s first explore the reasons to go for smaller stories. As mentioned above, there is often a felt need to come up with “small” user stories, whatever “small” might mean. This is often even part of making a requirement “ready” for the team to start working on it (cf. also how a felt need for “small” stories is a hint for immature teams).

The main reason behind moving from “big” feature requirements to “small” user stories typically boils down to the team wanting to ensure that the work they take on can be finished within one iteration. From the agile point of view, that’s a good idea: We want to deliver value in terms of working software as quickly as possible. But even if your delivery is not exactly continuously and also doesn’t always happen at the end of the iteration, there is a lot of value in using small stories: developers, UX, product owner, stakeholders should give and take feedback on the completed work to ensure the development is on the right track (in terms of “Did we make it right?” and validation “Did we make the right thing?”). The earlier we learn about problems, the better, so we can make easy adjustments. Arguably small stories can be implemented more quickly and thereby help getting the necessary feedbacks quickly. Also, in case adjustments need to be made, changing a small story is hopefully easy (instead of changing a big requirement). Small user stories also help planning: for one thing, it is usually far easier to come up with an accurate and precise estimate if you’ve broken down a big problem (“epic”) into tiny pieces.

What is a reasonable small story?

This one has an easy answer: any requirement that the team feels can be surely finished within one sprint is reasonably small.

Does size equal effort?

Unfortunately, this answer isn’t particularly useful. It is simply taking the outlined main reason for small stories as a minimal acceptance criteria. If we take a step back, the size of requirement seems here to be directly bound to the effort it takes to implement it. It looks like a continuum: from “no implementation effort” to “takes the entire team the entire sprint to implement”.

But a story that takes the team the entire sprint to implement is almost certainly still too big. Think about it from a risk perspective: let’s assume your estimate is just a bit wrong (as in “too low”), e.g. you discover some previously unknown technical problem during the implementation. In this case you will not be able to finish the story within the sprint — from a work management perspective, you clearly want to avoid this situation, as you’ll run into all sorts of unwanted discussions (from relevant ones like when you get feedback to irrelevant management ones like what can be done to help the team deliver on time or to come up with better estimates).

Effect of story size on burn up chart

Some teams take a very harsh stance at this and demand a “ready” story should not take longer than one day (of implementation and testing). I don’t believe in such strict rules but think a good upper bound is “no longer than one week”. With story point estimation this recommendation might look strange, but the simple idea is that if you know your team delivers ca. 20 points per two-weeks iteration, a 13 point story is probably too big, whereas an 8 point story could be small enough.

Of course, you can use the same reasoning of risk and project management to demand even smaller stories: e.g., with a 20 point velocity of a team, and three stories with 8 story points each, how would you plan your next sprint? Commit to two stories (summing up to 16 points) and leave one as a stretch goal? If you start working on the stretch goal but are not able to finish it, your efforts and results will not be visible (for the stretch goal). Instead, if you could break down the stretch goal into a 3 and a 5 point story, you might be able to commit to more points. Also, your burn-up chart might look a lot smoother with smaller stories than with big stories, cf. figure 2. But beware: what we really want is quick feedback and delivery of valuable increments. If smaller stories help you with this, great, if not, you’re likely just gaming numbers or making management happy (cf. also Johanna Rothman on the value of burndown and burnup) .

Consider functionality, too

However, the size of a requirement could also be addressed differently. Boris Gloger (e.g. in his German book on agile estimation) argues that estimates should not be based on effort or risk but on functionality. From this perspective, the smallest reasonable story is one whose implementation will enable one functionality.

This looks somewhat circular, as we have now moved from one fuzzy term (size) to another (functionality), in particular as in todays complex systems “one” functionality could be composed of a lot of different pieces playing together nicely. But requirement work is all about defining functionality in terms of specific, detailed user activities or system behaviors and as such is arguably easier to grasp and figure out than some number of story points.

It is usually the best idea to look at the functionality from the user’s point of view, because this will most closely resemble the business requirement anyway. But note that, without going further into architecture considerations, you might want to define functionality per system / component if you can define and observe the expected behavior per component. The “observe” part here is important as it is implemented, observable behavior that you can give feedback on. For the very same reason of getting feedback from humans, defining functionality from the user perspective is more preferable.

Now that we have discussed what the smallest story in terms of functionality could be, this might not be the most reasonable choice in your context. As a made-up example, consider the feature of where you want to save some document in some system. An additional feature could be to save a document under a different name (“save as”). From the implementation point of view, the difference in functionalities might be so small that it’s very easy to implement both in one go, so the team might decide that splitting off the “save as” is not worth it.

Another example could be where one user visible functionality (e.g. sending an order) requires the interaction of two systems (e.g. sending the order from some web application to some ERP system). Considering just one system part (e.g. the web application sending) as a user story might not be reasonable — it cannot be tested alone, hence doesn’t allow for feedback and also doesn’t provide any business value without the other system part. I believe this is where the entire team needs to give its input and consider not only the system functionalities but also (all) the tasks and the what is the minimum that makes a product increment valuable.

So now we have two different boundaries: whereas size in terms of functionality provides us with a lower boundary for a requirement (“implement at least one user-visible functionality”), size in terms of development effort provides an upper boundary (“can be implemented by the team in at most one week of an iteration”).

Clarity is required

Now, I believe there is a third “size” of a requirement which is related to clarity or level of understanding. Recall that at the start of many projects (or product phases) you’ll have a lot of very tiny and detailed requirements and some “bigger” requirements which have not been explored (or engineered) yet (which is what I would call “feature ideas”). An example of the former would be a user story along the lines of “As an editor, I need a save-as button right next to the save button so that I can give a new name under which to save my document”, whereas the latter would be something along the lines of “As a user, I want to save my preferences”. Quite obviously, the former is quite “small” and “clear”: it already says that a button is needed and what the expected behavior is. “User preferences” instead is just a label which might contain a huge list of things that need to be made customizable by the user and unless we break down this list into details, this requirement needs to be considered “big”. Another way of looking at this “big” story is that it is certainly not “ready” for implementation, so it is surely not a reasonable small-enough story. But even for the much clearer “save-as” story, there might be some lack of clarity on how to give the filename or other aspects (e.g. technical issues). Again, obviously we have a continuum between very clear and completely unclear.

Unfortunately, clarity alone will not be enough to decide whether a requirement is “small” enough: you can get very clear requirements that consist of lots of functionalities and are way too big to be implemented within one iteration (never mind huge use case or detailed specifications). However, from the perspective of “can we start implementing this requirement”, clarity of what needs to be implemented is the key ingredient that distinguishes a requirement from being “ready” from others. Clarity is a prerequisite to breaking it down into different functionalities and come up with a reasonable estimate (although it’s often also true that trying to break a big unclear requirement into smaller bits and pieces to come up with good estimates will help getting more clarity). Turning this around, even if you have a story that is reasonably small in terms of effort, if the team is raising concerns about the certainty of the estimates, the team is probably not sure whether it understands the functionalities that needs to be implemented and hence the story is probably still not “ready for implementation”.

Summing up, this leaves us with three important aspects influencing the “right” size of a story: clarity, functionality and effort. In the next article, I’ll explore some rules of thumb to split a big requirement into smaller parts.

In a blog post on dependency inversion in Clojure I’ve discussed what this DI principle actually is about and the solutions Clojure offers to support it. There is one aspect that bugged me a little: For me, a fundamental challenge with DI in a language like Clojure is that you often have simple functions depending on simple other functions (simple here in contrast to protocols). In the article linked to above I discussed one way of resolving function dependencies by using an indirection over a service locator. However, in practice writing service locator lookups by hand was getting tiresome soon. So instead I decided to have some fun throwing together some macros that handling function signatures, which resulted in a small library.

Enter funsig

funsig shoots lower than Clojure protocols: it provides dependency management on a per-function level. What this means is simply that you can define a function signature with defsig and then provide implementations with defimpl. Implementations will depend on the signature. Let’s say we have some application code that depends on a printer function:

(ns my.onion)

(defn printer [string]
    (println string))

(defn print-account-multiplied [account multiplier]
    (let [result (* account multiplier)]
        (printer result)))

One might want more flexibility on how and where to print. In other words, one might want the application code (print-account-multiplied) to depend on an abstraction (printer) only and not on the concrete implementation as in this example.

Funsig allows you to inverse the dependency of the printer implementation. You would define the signature with defsig and have the appplication code depend on the signature like this:

(ns my.onion
    (:require [de.find-method.funsig :as di :refer [defsig defimpl]]))

(defsig printer [string])

(defn print-account-multiplied [account multiplier]
    (let [result (* account multiplier)]
        (printer result)))

You can then provide the implementation with defimpl:

(ns my.onion.simle-printer
    (:require [de.find.method.funsig :as di :refer [defimpl]]
              [my.onion :as mo :refer [printer]]))

(defimpl printer [string]
    (println string))

Note that the implementation has a dependency on the signature, not the other way around. Also, application code (print-account-multiplied) simply depends on the signature — here the signature is in the same file, but reference to the var in another namespace (i.e. using require\:refer) also works normally. For application code, this looks like dependency injection.

funsig will also allow you to have multiple implementations and then select the one you want.

Have fun!

You’ll find all the gory details in the README and / or in the intro document. The latter also explains the relationship to the service locator pattern mentioned above.

Feedback welcome!

Technical debt assumes that you have an existing system and you know already about the areas where the duct tape is becoming thin. Let’s discuss the problem, the conflict around technical debt a little. It’s basically always the same battle: the guys maintaining the system probably have good reasons why they want to pay back on the accumulated technical debt, whereas the product owner believes more functionality is more important.

This is a typical clash of interests between people from different tribes and usually from different departments: the developers report to a technical manager, the product owner to a business guy. The IT manager is typically under pressure to minimize costs, in particular costs for support and maintenance, so he’s interested in paying back on technical debt during projects. The business manager, however, is under pressure to convince new clients with new features. So, that there is a difference in how technical and non-technical people set priorities between technical debt and new features doesn’t come as a surprise: naturally, to the business side this is all about technical issues which they don’t consider to be their concern. It’s only natural that they believe that tackling such technical issues should be solved by the technical folks “somehow”. After all, it were these guys which build the system with all these problems in the first place, and now they want to spend more time and money on this? Although I’ve exaggerated here a little maybe and this line of argument is way too simplistic (given that often debt is accumulated over time), it’s still quite popular. Such product owners seem to believe that they are only responsible for developing new functionality. Corporate culture can contribute to this: some companies have a culture where every new glitter is taken for a star and gets way more attention than the cash cow functionality that keeps existing clients happy.

So some product owners will see technical debt as a separate issue which needs to go on a maintenance budget, a budget that somebody else is responsible for, e.g. the IT department. The problem with that idea is that in a culture where building new features is the only thing that counts, typically no one ever gets around to clean up the technical debt. All what will happen is that you’ll have some people that fix bugs with this maintenance budget. So you don’t actually ever spend the money on technical debt but only on the results of it which of course doesn’t address the root of the technical issues. Also, if you finally do get an approval for a refactoring project, these are really the most boring and horrible types of projects to work on. So it’s not likely to see a lot of happy, motivated people on such projects — and sure enough the most competent people are probably assigned to more valuable projects anyway. As a result and because there is no business pressure on such refactoring projects, as soon as something else comes up, such refactoring initiatives are abandoned or at least down-sized to a minimum. Peter Seibel describes in a recent, lovely written article the need for an Engineering Effectiveness team at Twitter in which he lays out in some details how hard it is to really put up with technical debt.

Let’s drill down on technical debt from the point of view of requirements. Wait a minute, this doesn’t sound right — how could technical debt be a requirement? Obviously, it isn’t, but there is a close relation. Technical debt can come in different flavors: e.g. you really should install a second machine and a load balancer so you’re prepared for failure or we really need to rewrite module foobar.clx to finally get rid of all the spaghetti code that’s slowing us down to oh, we really need to support responsive design, we’re getting more mobile users every day. Now, if you take a look at these three simple examples, they are related to quality requirements: the first is about availability, the second about maintainability and the last about usability. Take a look at the ISO 25000 standard for software product quality page listing all the quality attributes one might want to consider. Technical debt is always tied to some quality that your system should offer, but doesn’t.

A reason for this is that quality attributes are often not made explicit during requirements gathering, regardless of whether you’re following an agile approach or not. The less technical the product owner, the more often they seem to assume that performance, scalability, security and the other -ilities will come out just right by magic. Technicians, on the other hand, know perfectly well that they will not. It’s worse if they don’t that: they will build something that might fulfill the functional requirements but not the non-functional requirements. “Works on my machine” might be fine for a naive developer, but nobody will be happy if takes 5 minutes to load the page for the gazillions of parallel requests in production. There is a reason why software architecture is mostly concerned with quality, if you don’t plan and build for scalability, it’s unlikely you end up with a highly performing system. Take a look at the picture: this bridge wasn’t planned to cope with the amount of traffic it sees nowadays, some time in the 80s somebody decided it would be okay to have three lanes instead of two and didn’t think through the long term consequences.

The bridge of the highway A40 over the Rhein in Duisburg. It is damaged so bad that trucks are no longer allowed to cross it.
Picture made by kaʁstn Disk/Catstitched by Daniel Schwen, licensed under CC BY-SA 3.0 de over [Wikimedia Commons]( https://commons.wikimedia.org/wiki/File:Rheinbruecke_Neuenkamp_pano.jpg#/media/File:Rheinbruecke_Neuenkamp_pano.jpg)

Of course, technical debt can also accumulate over time, one might rightly point out: sure, one machine was enough for the requirements at the start, but nobody followed up on the increase in users and nobody cared (or had time) to clean up the code in module foobar.clx back then (nor in the following six years of quick bug fixes and minor one-off patches that have grown on it like leeches). This is a sign that nobody actively had an eye on how the world in and around the systems changes and to take action early on. For code quality, Uncle Bob points to the Boy Scout rule which says that you should always clean up (regardless of who messed it up) — another way of trying to ease maintainability by paying back on technical debt every day.

The over-aching point here is that you, as a technician and you, too, as a product owner need to think about the quality requirements of your system and to do that over the entire life cycle. What was a fitting solution at a time might turn into technical debt over time. This means means your system now doesn’t hold up to the quality requirements you and your clients needs and no matter what was the cause for it, you’re better of fixing it now than accumulate even more technical debt.

Isaac Sacolick describes in his article on How to get an agile product owner to pay for technical debt how to address the common problem of technical debt heads on, mostly from a managers perspective via process and by making people (mostly the technical lead) responsible. However, I think it makes a lot more sense to try to ensure a common understanding between the developers and the product owner. One way to go about this is to use quality scenarios: as a member of the dev team, ask your product owner how many users the system needs to handle. Ask her if she thinks it’s okay if somebody might find a security issue earlier than you. Or if she thinks it’s okay when a seemingly small change will take three weeks because the code is an intangible mess that nobody understands since Dieter left? These sort of questions hopefully open up to discussions based on business value, ROI and cost of delay. You are then using the language product owners understand and you might also learn something along the line (no we expect so many users to do X in the future, so investing in module Z doesn’t make sense).

Now, granted that doesn’t always work. An old colleague of mine always denounced the old system he worked on as being exactly the way it was ordered. Isaac’s article might offer some advice here, of which the most important one is probably that paying back technical debt is way easier if you have a CIO or someone with similar power supporting you.

I’m happy that there are apparently still people writing articles about Scrum that are not ranting and venting about how agile sucks: Colin Higgins explains that Scrum is not our saviour. It basically boils down to the old no silver bullet saying:

No, SCRUM, or any of the other agile methodologies will not just magically fix your throughput problems after you adopt every facet of them religiously. These methodologies will just show you how badly your development practices suck.

I agree completely. But there are some agile development practices or tools which can help you. Go look up extreme programming (XP) and what it urged developers to do. CI/CD, TDD and pair programming are by now widely accepted development practises which can help, because they aim at bringing up problems way before they end up in production.

However, there is a boundary to what these tools alone can offer. A lack of knowledge of the technologies or a general lack of problem solving skills are fundamental issues which you can overcome only by learning and gaining experience, if at all. Another quote from the linked article:

Agile process alone won’t fix your development problems, you need technical excellence for successful projects. The two are necessary for a continually successful organization.

I would agree in principle, but there are two important issues here:

1) “A fool with a tool is still a fool” and “technical excellence” are the two extreme ends of a large scale. It’s the same as with car driving: most developers like to see themselves among the top 5% of drivers developers, but as we all know there is not enough room for everyone within the 5%. Still, too many developers would shrug the call to technical excellence away, because they believe they are so great or they have no idea what technical excellence actually means. For these developers, there is always someone else to blame.

2) Requiring the proverbial “rock star developer” is — for reasons discussed above — in my opinion not a viable option. In other words, while Scrum might not be a saviour, we’re not doing anybody a favor if we instead declare “technical excellence” as the silver bullet that can lead to success. Yes, we should strive for technical excellence and do our best to improve our capabilities (technical and otherwise). This is what the article rightfully demands of developers: do the best you can and try to continuously improve. But there still needs to be a middle ground, where it’s possible to run projects / products successfully with the people you currently have. Agile helps with that because it makes clear what is realistically possible and it aims for continuous improvement, e.g. with retrospectives. To repeat the quote from above: “These methodologies will just show you how badly your development practices suck.” That’s a great starting point. It’s a learning opportunity. Go use it.

Note that this is important not only on a personal level, but also on an organizational level: Scrum Masters should try to change the company culture to value learning. This requires room for experiments and for failures, where room translates to time and safety (i.e. support). On the sprint level, this might translate to using spikes, but on an organizational level it might involve things like prototypes, split testing and other ideas from the lean startup camp.

For a recent project, I need to extract data from a sqlite3 database. Writing the Clojure code to retrieve data was very straight-forward with clojure.java-jdbc and java-jdbc/dsl. Naturally, I wanted to have some tests for this code as well. In a previous Python project, I had a lot of fun using sqlite’s in-memory feature to run very speedy database tests, so of course I wanted this for my current Clojure project, too. This turned out not to be so easy as I had expected though, so I’m documenting it here for the next naive soul. My initial attempt with clojure.java.jdbc, java-jdbc/dsl and midje looked basically like this:

    (def testdbspec
      {:subprotocol "sqlite"
       :subname ":memory:"})

    (defn make-bookmark-table []
      (jdbc/with-transaction [db testdbspec]
         (jdbc/db-do-prepared db
           (ddl/create-table :bookmarks
                      [:id :int :primary :key]
                      [:type :int]
                      [:title "longvarchar"]))))

    (defn add-bookmark []
      (jdbc/with-transaction [db testdbspec]
         (jdbc/db-do-prepared db
           (str 
              "INSERT INTO bookmarks (id, type, title) "
              "VALUES ('12453', '2', 'a bookmark')"))))

    (defn setup-database [db]
       (make-bookmark-table)
       (add-bookmark)

    (facts "Testing database access to bookmarks"
       (with-state-changes [(before :facts (setup-database))]
            (fact "We can retrieve a list of bookmarks"
                (fetch-tags :dataspec testdb) => [{:title "a bookmark"}]))))

This will fail quite early, because basically as soon as the with-transaction in make-bookmark-table has finished its work, the connection to the database will be closed. As a result, when the next with-transaction or jdbc\query is run, you’ll connect to a fresh in-memory database which doesn’t have the tables you just created. My old Python test code didn’t have this problem, because the setUp method of the TestCase would create the database connection (via sqlalchemy’s create_engine) and would keep it alive until the TestCase tearDown method would run.

I tried giving back the database connection from make-bookmark-table, but this just results in a “connection closed” error. Unfortunately, clojure.java.jdbc doesn’t support opening and closing the connection yourself. Sure, you can use get-connection, but you can’t feed this into either with-transaction or query. query uses with-open internally, which will conveniently close the connection for you. In a post on the perils of dynamic scope Stuart Sierra calls this the Dynamically-Scoped Singleton Resource and files it under ‘anti-pattern’. I gotten bitten pretty exactly by what Stuart describes: when dealing with sqlite’s in-memory feature, we would like to manage the connection ourselves, but we can’t.

After banging my head against this for a while, the only option I could come up with resorts to extract the relevant with-transaction from the setup code. Instead you have to wrap the tests with the transaction and then call the setup code, like this:

    (defn make-bookmark-table [db]
      (jdbc/db-do-prepared db
           (ddl/create-table :bookmarks
                      [:id :int :primary :key]
                      [:type :int]
                      [:title "longvarchar"])))

    (defn setup-tables [db]
       (make-bookmark-table db))

    (defn add-bookmark [db]
       (jdbc/db-do-prepared db
           (str 
              "INSERT INTO bookmarks (id, type, title) "
              "VALUES ('12453', '2', 'a bookmark' )")))

    (defn remove-bookmark [db]
       (jdbc/db-do-prepared db
            (str "DELETE FROM bookmarks WHERE id = '12453")))

    (facts "Testing database access to bookmarks"
       (jdbc/with-db-transaction [db testdbspec]
            (setup-tables db)
            (with-state-changes [(before :facts (add-boomark db))
                                          (after :facts (remove-boomark db))]
                 (fact "We can retrieve a list of bookmarks"
                     (fetch-tags :dataspec db) => [{:title "a bookmark"}]))))

This works as expected.

The problem

I was recently reading a nice German book on Effective Software Archictecture by Gernot Starke and stumbled upon a discussion of the dependency inversion principle, which got me thinking. Gernot Starke first discusses the problem with an allusion to traditional procedural programming (translation mine):

Classical designs for procedural languages show a very characteristic structure of dependencies. As (the following figure) illustrates, these dependencies typically start from a central point, for instance the main program or the central module of an application. At this high level, these modules typically implement abstract processes. However, they are directly depending on the implementation of the concrete units, i.e. the very functions. These dependencies are causing big problems in practice. They inhibit changing the implementation of concrete functions without causing impacts on the overall system.

He then goes on to introduce the idea of programming against abstractions and introduces the idea of the dependency inversion principle, first coined in Bob Martin’s DIP article (see also another thorough discussion in Brett Schucherts article on DIP). Basically, the idea is that the integrating process refers only to abstractions (i.e. interfaces) which are then implemented by concrete elements (classes), cf. the next figure.

When I take a look at some of my recent Clojure code or at some older code I’ve written in Common Lisp, I immediately recognize dependencies that correspond to those in a classical procedural system. Let’s go for an example and take a look at one specific function in kata 4, data munging:

(ns kata4-data-munging.core
    (:require [kata4-data-munging.parse :refer [parse-day]]
              [clojure.java.io :as 'io]))

(defn find-lowest-temperature
    "Return day in weatherfile with the smallest temperature spread"
    [weatherfile]
    (with-open [rdr (io/reader weatherfile)]
         (loop [lines (line-seq rdr) minday 0 minspread 0]
        (if (empty? lines)
            minday
            (let [{mnt :MnT mxt :MxT curday :day} (parse-day (first lines)) ;<-- dependency!
              curspread (when (and mnt mxt) (- mxt mnt))]
            (if (and curday curspread
                  (or (= minspread 0)
                  (< curspread minspread)))
               (recur (next lines) curday curspread)
               (recur (next lines) minday minspread)))))))

The dependency here is on the concrete implementation of parse-day, you can basically ignore the rest for the argument here. Given that this was a small coding kata, this is not unreasonable (and in the course of the kata, the code changes to be more general), but the issues here are obvious:

• if we would like to parse a weather-file with a different structure, we have to change find-lowest-temperature to call out to a different function,
• if the result of the new function differs, again we have to change the implementation of find-lowest-temperature,
• we also have to change the namespace declaration, i.e. we probably want to require a different module.

Clojure’s built-in solutions

The application of the dependency inversion principle is typically shown in the context of object-oriented programming languages, like Java where you use interfaces and classes implementing those interfaces for breaking the dependency on concrete implementations, cf. the figure above again. But as we’ll see the principle can be applied independently of object-orientation. I’ll discuss higher-order functions, protocols and multimethods as potential solutions.

Higher order functions

For starters and probably painfully obvious is to make use of the fact that Clojure treats functions as first-class objects and supports higher-order functions. This simply means that we can pass the parsing function as an argument to find-lowest-temperature.

(defn find-lowest-temperature
    "Return day in weatherfile with the smallest temperature spread"
    [weatherfile parsefn] ; <-- function as parameter
    (with-open [rdr (io/reader weatherfile)]
         (loop [lines (line-seq rdr) minday 0 minspread 0]
        (if (empty? lines)
            minday
            (let [{mnt :MnT mxt :MxT curday :day}  (parsefn (first lines))
              curspread (when (and mnt mxt) (- mxt mnt))]
            (if (and curday curspread
                  (or (= minspread 0)
                  (< curspread minspread)))
               (recur (next lines) curday curspread)
               (recur (next lines) minday minspread)))))))

This way, we can simply call (find-lowest-temperature "myweatherfile" parse-day) and freely substitute whatever file format and accompanying parse function we need. What does this buy us?

• We no longer have to modify find-lowest-temperature when we want to use a different parse-day function.
• The namespace containing find-lowest-temperature also no longer requires the (namespace containing the) parse function.

But there is also a down-side: find-lowest-temperature assumes that all parsing functions it will get fed adhere to a signature that is entirely implicit: parsefn needs to take exactly one line and needs to return a map with given key-names. Higher-order functions don’t provide a solution for this per-se, so in order to solve the implicit signature issue we need to look elsewhere. This is nothing Clojure specific: Assuming you’ve passed in an object either as a method parameter or via Setter-Methods or Constructor-Injection (cf. dependency injection), Python’s or Ruby’s duck-typing basically works the same way: the caller of a method simply assumes that the callee offers a method with the right signature. It is the responsibility of the caller (of find-lowest-temperature) to provide a matching function for parse-fn.

However, this actually amounts to just move the problem from one level to the next: now some other level has to decide which concrete parse function needs to be used. This next level will have again the exact same problems: it will depend on both concrete implementations of find-lowest-temperature and parse-day (or any other parse function). If you think this through, it’s obvious that in general at one point or another, you have code that determines which function to call and which parameters to use. The question is only if we can use abstractions or whether we have to use concrete implementations. We’ll return to this issue, that now at some other level you need to handle the problem, later.

Kata 14 is a seemingly simple one that is concerned with, as Dave Thomas puts it, “the heuristics of processing” texts, using trigrams to produce (more or less) random new texts.

Trigrams are not a new concept for me. Although the underlying concept is simple, they can be used for many interesting applications. Trigrams are a special case of N-grams, where N=3 turns out to be especially useful (as in “giving better results as other values for N”) for natural language processing, at least for western languages. Nearly a decade ago, I had the pleasure to collaborate with some rather smart people who used trigrams to identify “matching” text snippets between dictionary entries. The idea was similar to what is described in this article on using trigram vectors and nearest neighborhood calculation for document classification.As I’m generally interested in NLP and not only in doing coding katas, I will mainly focus on the trigram aspects in this kata, not so much on the random text generation part.

If you followed the link to the Wikipedia article, it’s clear from the kata description that we need word-level trigrams, not character-level trigrams. The kata description also already reveals the data structure to use for solving the task, a HashMap and the algorithm is also described in enough detail to be straight-forward.

Let’s augment the kata a little and decompose the tasks:

1. split some string into n-grams with default n=3, where we might want to apply different criteria to apply on where we can split the string (e.g. after each character or after each word)
2. parse a file into n-grams, where we need to consider sentence boundaries
3. parse a collection of files into n-grams concurrently (just to speed up parsing of a larger file collection and also to introduce another possibility to learn a little more about Clojure’s specific tools to handle concurrency)
4. do some analysis on the trigrams found in the recommended Tom Swift and his aircraft text
5. modify the n-gram computation to yield the “first two words as key / list of all third words value” map described in the kata
6. build a lazy-seq version of the text generation algorithm (because, as the example in the description already shows, there might be circles which could lead to infinite results)
7. maybe implement the nearest neighborhood classification scheme described in the paper linked just for fun

But first things first: let’s parse some string into trigrams. This, first of all, requires tokenization. As a first obvious naive idea, we start out with simple string splitting, using clojure.string. First let’s split on all whitespace #"\s", using the first sentence in the Tom Swift text:

kata14-trigrams.core> (str/split "Are you all ready, Tom?" #"\s")
["Are" "you" "all" "ready," "Tom?"]

This already shows the issues surrounding punctuation that Dave Thomas mentions in the kata description. Basically, we have to consider what we want to do with sentence boundaries. Fortunately, we’re ultimately using Java’s Pattern class, so we can also match (or split) on punctuation, although probably not on all punctuation, but only on those which signify a sentence boundary (i.e. the charset [.!?] followed by either whitespace #"\s+" or end of line $):

kata14-trigrams.core> (str/split "Are you all ready - Tom?" #"\s*\p{Punct}\s*")
["Are you all ready" "Tom"]
kata14-trigrams.core> (str/split "Are you all ready, Tom?" #"\s*[.?!](\s+|$")
["Are you all ready, Tom"]
kata14-trigrams.core> (map #(str/split %1 #"\s+")
                      (str/split "Are you all ready, Tom? I want to go."
                        #"\s*[.!?](\s+|$)"))
(["Are" "you" "all" "ready," "Tom"] ["I" "want" "to" "go"])

This still leaves the question open of what we want to do with the comma or any other interleaving punctuation. It’s clear that we want to get rid of it somehow, but it’s not too clear whether we would like to see “Tom” as a valid consecutive element in the text generation part. Probably not, so an idea here would be to try to make the remaining punctuation elements visible as separate tokens.

Let’s put this issue aside and move on to the actual n-gram generation. Quite obviously, “computing” an n-gram is really just a simple sequence operation: you move through the sequence, always taking n elements as needed, until you’re done. This is completely straight-forward to accomplish with a simple accumulator (acc) to collect the results that we take while looping through the sequence. (Code is on github, as always.)

(defn ngram
   "Given a sequence sq and a number n, returns a sequence of new contiguous sequences of n items that appear in sq."
   ([squence n]
     (ngram squence n []))
   ([squence n acc]
         (if-let [sq (seq squence)]
             (recur (rest sq) n (conj aux (take n sq)))
             acc)))

Given that we might want to run this on longer strings (texts, books), it makes sense to make this lazy by wrapping the call to the accumulator version in a lazy sequence.

 kata14-trigrams.core> (ngram [1 2 3 4 5 6] 3)
 ((1 2 3) (2 3 4) (3 4 5) (4 5 6) (5 6) (6))
 kata14-trigrams.core> (realized? (ngram [1 2 3 4 5 6] 3))
 false
 kata14-trigrams.core> (take 2 (ngram [1 2 3 4 5 6] 3))
 ((1 2 3) (2 3 4)

Okay, let’s combine this with our clojure.string/split experiments:

kata14-trigrams.core> (map #(str/split %1 #"\s+")
                      (str/split "Are you all ready, Tom? I want to go."
                        #"\s*[.!?](\s+|$)"))
(["Are" "you" "all" "ready," "Tom"] ["I" "want" "to" "go"])
kata14-trigrams.core> (map #(ngram %1 3) *1)
((("Are" "you" "all") ("you" "all" "ready,") ("all" "ready," "Tom") ("ready," "Tom") ("Tom"))
 (("I" "want" "to") ("want" "to" "go") ("to" "go") ("go")))

Okay, this looks like we’ve basically have everything we need in hand, now let’s make it a little bit more formal. First of all, the tokenization step. So far, we have basically done two things in one step, sentence boundary detection and in-sentence tokenization. I already hinted at the need to do further work on the in-sentence tokenization wrt. punctuation and there might be other steps that we might want to add in the future, for instance, stemming or further morphological analysis. I’ll not go in the direction of a more thorough tokenization method, which would require to go beyond regular expressions for many languages, but at least let’s communicate the intention of how the tokenize method works clearly.

(fact "Tokenize an input string, splitting sentences along the way"
      (tokenize "Are you ready, Tom? I want to go.") => '(("Are" "you" "ready" "," "Tom")
                                                                  ("I" "want" "to" "go")))

(defn tokenize
  "Tokenize a string"
  [string]
  (-> string
     (split-sentences)
     (tokenize-sentences)))

This is basically the top-level function for tokenizing an incoming string, threading the result of splitting sentences into a tokenization function. Let’s take a look at the details, which shouldn’t be surprising at all. First we have split-sentences, which is basically str/split on sentence end markers. Then we have split-on-whitespace, which we’ve also already seen. split-off-punctuation is basically handling all punctuation not used up during sentence boundary detection, which we will want to keep. And then we have two wrappers tokenize-sentence(s) which do nothing more than handling mapping over the various bits and pieces. This concludes the tokenization step, phew.

(fact "Split sentences in a string"
      (split-sentences "Are you ready, Tom? I want to go.") => ["Are you ready, Tom" "I want to go"])

 (defn split-sentences
   "Split a string into a sequence of sentences"
   [string]
   (str/split string #"\s*[.!?](\s+|$)"))

    (fact "Splitting on whitespace"
     (split-on-whitespace "Are you  ready") => ["Are" "you" "ready"]
     (split-on-whitespace "Are") => ["Are"])

(defn split-on-whitespace
  "Take a string and split it's content on whitespace, removing the whitespace"
  [string]
      (str/split string #"\s+"))

(fact "Splitting but keeping punctuation if any"
     (split-off-punctuation "ready,") => ["ready" ","]
     (split-off-punctuation "ready") => ["ready"]
     (split-off-punctuation "!+#?") => [""])

(defn split-off-punctuation
   "Take a string and split it's content, keeping punctuation as new tokens"
  [string]
  (let [match (re-find #"(\w+)(\p{Punct})?" string)
            result (rest (keep identity match))]
     (if (seq result)
    result
    (vector ""))))

(fact "Tokenize a sentence"
      (tokenize-sentence "Are you ready, Tom?") => '("Are" "you" "ready" "," "Tom" "?"))

(defn tokenize-sentence
  "Take a single sentence and return a sequence of tokens for it"
  [sentence]
  (flatten (map split-off-punctuation
               (split-on-whitespace sentence))))

(fact "Tokenize some sentences"
       (tokenize-sentences ["Are you ready, Tom?" "I want to go."]) => '(("Are" "you" "ready" "," "Tom" "?")
                                                                         ("I" "want" "to" "go" ".")))

(defn tokenize-sentences
   "Take a sequence of sentences and return a sequence of tokens for  each sentence"
  [sentences]
  (map tokenize-sentence sentences))

When combining this with the ngram function, this is already pretty close to what we’ll need to solve the original kata, although we will need some further adjustment to the data structure, which I’m going to tackle later.

kata14-trigrams.core> (map #(ngram %1 3)
                       (tokenize "Are you ready, Tom? I want to go."))
((("Are" "you" "ready") ("you" "ready" ",") ("ready" "," "Tom") ("," "Tom") ("Tom"))
(("I" "want" "to") ("want" "to" "go") ("to" "go") ("go")))

So, let’s move to the next part which is generating ngrams for an entire file. First of all, I think that again we better do this in a lazy fashion, no need to do lots processing of huge files that might not be completely necessary. Looking back we can see that for any given string tokenize processes the same string at least thrice: we’re first splitting on sentence boundaries, then handle punctuation and finally split on whitespace. If you think about reading files, it’s quite obvious that the readLine method of java.io.BufferedReader which is behind Clojure’s line-seq is also processing buffers quite similarly looking for line ends to split on. Maybe, we can combine some of the work? Let’s start out with figuring out how to process a file char by char lazily. An answer to a stackoverflow question on processing files per character in Clojure strictly follows line-seq:

(defn char-seq
  [^java.io.Reader rdr]
   (when-let [chr (.read rdr)]
      (if (>= chr 0)
          (cons (char chr) (lazy-seq (char-seq rdr))))))

This is a start but not too helpful as discussed in this other stackoverflow thread on processing large text files, as the result of line-seq and char-seq is a cons and the lazy part of it doesn’t help you much when you’re not processing the file right away. Instead one might want to return a lazy sequence of results, closing the file only afterwards. This could like this:

; cf. https://stackoverflow.com/questions/4118123/read-a-very-large-text-file-into-a-list-in-clojure/10462159#10462159
(defn lazy-file-chars [file]
   (letfn [(lfl-helper [rdr]
              (lazy-seq
               (if-let [chr (.read rdr)]
                      (when (> chr 0)
                          (cons (char chr) (lfl-helper rdr)))
                      (do (.close rdr) nil))))]
      (lfl-helper (clojure.java.io/reader file))))

When you look at this simple piece of code, besides reading characters from disc and building up a lazy-seq, it’s also a) doing a sanity check on the input and b) building up a particular structure to return. Sounds exactly like the hooks we might want to consider for parsing sentences on read. Let’s rip the code apart and combine it with the guts of split-sentences (matching explicitly on characters instead of using regular expression character classes):

 (defn read-next-sentence [rdr aux]
    (if-let [chr (.read rdr)]
       (let [character (char chr)]
         (cond (= \. character) aux
                       (= \? character) aux
                       (= \! character) aux
                       (= \tab character) (recur rdr (conj aux \ ))
          :else (recur rdr (conj aux character))))
    aux))

(defn file-sentences [file]
     (letfn [(lfs-helper [rdr]
                 (lazy-seq
               (if-let [sentence (seq (read-next-sentence rdr (vector)))]
                 (cons (apply str sentence) (lfs-helper rdr))
                 (do (.close rdr) nil))))]
                (lfs-helper (clojure.java.io/reader file))))

read-next-sentence has some obvious deficiencies: it now splits sentences on every occurrence of .?!, not only on those occurrences which are followed by whitespace. Second, it should handle (only) multiple occurrences of \return\newline characters (CRLF) as sentence delimiters, too. Solving both of these issues requires to go in the direction of real parsers where we would have to see aux as a stack of previously read characters. And we might not only want to deal with tabs specially (turning them into a space), e.g. we might want to replace multiple spaces/tabs into a single space etc. I’ll just draw a sketch here that we might want to elaborate further:

(fact "Test for sentence end"
      (sentence-end-p \space   [\g \o \.])                    => true
      (sentence-end-p \space   [\r \e \a \d \y \?])           => true
      (sentence-end-p \newline [\y \return \newline \return]) => true
      (sentence-end-p \newline [\y \newline])                 => true
      (sentence-end-p "B"      [\.])                          => false
      (sentence-end-p \newline [\y \return])                  => false
      (sentence-end-p \newline [\y])                          => false)

   (fact "Parse result for characters depends on previous reads"
      (next-char-result \space   [\g \o \space])  => [\g \o \space]
      (next-char-result \tab     [\g \o])         => [\g \o \space]
      (next-char-result \tab     [\g \o \space])  => [\g \o \space]
      (next-char-result \tab     [\g \o \tab])    => [\g \o \space]
      (next-char-result \return  [\g \o])         => [\g \o]
      (next-char-result \newline [\g \o \return]) => [\g \o \space]
      (next-char-result \newline [\g \o])         => [\g \o \space])

(defn sentence-end-p [character charstack]
    (cond (and (= character \space)
                       (some (partial = (peek charstack)) [\. \? \!])) true
                (and (= character \return)
                       (some (partial = (peek charstack)) [\. \? \!])) true
                (and (= character \newline)
                       (some (partial = (peek charstack)) [\. \? \!])) true
                (and (= character \newline)
                       (or (= (peek charstack) \newline)
                           (and (= (peek charstack) \return)
                                (= (peek (pop charstack)) \newline)))) true
                :else false))

(defn next-char-result [character charstack]
     (cond (and (empty? charstack)
                (or (= character \space)
                                (= character \tab)
                                (= character \newline)
                                (= character \return))) charstack
                 (and (= character \space)
          (= (peek charstack) \space))  charstack
         (and (= character \tab)
          (= (peek charstack) \space))  charstack
             (and (= character \tab)
              (= (peek charstack) \tab))  (conj (pop charstack) \space) ; should never happen
          (= character \tab) (conj charstack \space)
                      (= character \return) charstack
                 (and (= character \newline)
                      (= (peek charstack) \space)) charstack
                 (and (= character \newline)
                      (= (peek charstack) \return)) (conj (pop charstack) \space) ; should never happen
                      (= character \newline) (conj charstack \space)
                 :else (conj charstack character)))

I’ll leave it at that, although it’s clear that we can and probably should extend it in many different ways. Here are the adapted functions to use these:

(defn read-next-sentence
   ([rdr]
       (read-next-sentence rdr (vector) (vector)))
   ([rdr seen result]
       (let [chr (.read rdr)]
           (if (and chr
                    (>= chr 0))
               (let [character (char chr)]
                   (if (sentence-end-p character seen)
                    result
                        (recur rdr (conj seen character)
                           (next-char-result character result))))
               result))))

(defn read-sentences [x]
    (letfn [(lfs-helper [rdr]
                (lazy-seq
                (if-let [sentence (read-next-sentence rdr)]
                    (cons (apply str sentence) (lfs-helper rdr))
                    (do (.close rdr) nil))))]
        (lfs-helper (clojure.java.io/reader x))))

The result is here that we now have a read-next-sentence function which just reads (non-lazily) and a (local) helper function which uses it to build up lazy sequence of sentences. Let’s test it briefly:

kata14-trigrams.core> (pprint 
                     (map #(ngram %1 3) 
                  (tokenize-sentences 
                  (take 2 
                        (read-sentences test-file)))))
((("The" "Project" "Gutenberg")
 ("Project" "Gutenberg" "EBook")
 ("Gutenberg" "EBook" "of")
 ("EBook" "of" "Tom")
 ("of" "Tom" "Swift")
 ("Tom" "Swift" "and")
     ("Swift" "and" "his")
     ("and" "his" "Airship")
  ...

Although one would probably now integrate more functionality from tokenize-sentences into read-next-sentence, I’ll won’t elaborate this now and see task 2 as solved. As a side note, this looks as if it’s only restricted to files now, but it really isn’t, as clojure.java.io/reader will happily accept StringReader arguments:

kata14-trigrams.core> (import java.io.StringReader)
java.io.StringReader
kata14-trigrams.core> (take 2 (read-sentences (StringReader. "This is a sentence. And another one")))
("This is a sentence." "And another one")

Christophe Rhodes’ post on http-content-negotiation and generalized specializers in CLOS (Common Lisp Object System) made an ugliness in a small Clojure web application jump right into my face. I’m using liberator to setup so-called resources (side-note: While this post assumes some familiarity with liberator, the main aspects is actually multi-method handling in Clojure — I hope it’s useful even if you don’t know or care about liberator). Resources are serving as ring handlers (typically used with compojure) and are used to deal with most aspects of request handling in a fairly declarative manner, including content negotiation. Liberator provides decision points and handlers, moving a so-called context around between the various functions that you need to associate with resources — map-like data returned from a decision function will be merged with the existing context. So far, so good. The bad part, however, was that I used a single resource definition for providing multiple media types. More exactly my code / resource definition has an anonymous handler function which uses a simple value check to serve the correct media type (we’re talking about the Accept-Header of the incoming request, cf. RfC 2616, Sec. 14.1, like this (as you can imagine, that’s a somewhat simplified version):

 (defresource users
   :available-media-types ["text/html" "application/json"]
   :method-allowed? (request-method-in :get)
   :exists? (fn [context]
                {:users (find-users)})
   :handle-ok (fn [context] 
                  (let [media-type
                       (get-in context [:representation :media-type])]
                    (condp = media-type
                        "application/json"
                          (generate-string (get context :users))
                        "text/html"
                          (usersview (:users context)))))
   :handle-not-found (fn [context]
                         (let [media-type
                              (get-in context [:representation :media-type])]
                            ;; TODO: Handle not found for HTML
                            (condp = media-type
                                "application/json"
                                   (generate-string {:error "No such user"})))))

From a functional point of view, there is not much wrong with this. It’s very close to the description in the relevant part of the liberator tutorial on content negotiation. But from an aesthetical point of view, the condp expressions to determine finally how to present the resource data is plainly ugly. To get rid of this ugliness, the inspiration I took from Christophe’s article is to rely on Clojures method dispatch (which is the simple part from Christophe’s post only). The idea is straight-forward: Instead of using a simple anonymous function which convolutes two different media-types, introduce a multi-method like users-handle-ok that dispatches on media-type: We can simply define a dispatch method (via defmulti), moving the code which determines the accepted media-type (e.g. “application/json”). This value is then used by Clojure to determine the right method to use.

(defmulti users-handle-ok 
  "Handle OK for users resource for different media-types"
  (fn [context]
  (get-in context [:representation :media-type])))
(defmethod users-handle-ok "application/json" [context]
  (generate-string (get context :users)))
(defmethod users-handle-ok "text/html" [context] 
(usersview (get context :users)))

;; some code elided here ...

(defresource users
  :available-media-types ["text/html" "application/json"]
  :method-allowed? (request-method-in :get)
  :exists? #(users-exists? %)
  :handle-ok #(users-handle-ok %)
  :handle-not-found #(users-handle-not-found %))

From a clean code perspective, this has two benefits: we now have mainly code left which does one thing at a time (SRP), which is what we should aim for and which makes unit testing also somewhat easier and more to the point. It also slims down the amount of code in the resource definition considerably. It’s now much more obvious that the resource definition is (from the application developer point of view) not much more than an integration point for different other functions.

Of course, we can use a similar approach for all of the other methods as well. Let’s assume that I have a resource that can generate HTML and JSON, but expects that all incoming POST requests contain JSON only. This will look utterly similar to the approach above, only this time we dispatch on the request-method. If we are now POST-ing to this resource with a different Content-Type, we’ll receive a “415 Unsupported media type” reply from liberator.

(defmulti known-content-type?
  "Determine known content types depending on request-method"
  (fn [context]
  (get-in context [:request :request-method])))
(defmethod known-content-type? :post [context]
  "Allow only application/json for POST requests"
  (when-let [content-type (get-in context [:request :content-type])]
    (condp = content-type
           "application/json" true
           false)))      
(defmethod known-content-type? :default [_]
  true)

(defresource someresource
  :available-media-types ["text/html" "application/json"]
  :method-allowed? (request-method-in :get :post)
  :known-content-type? #(known-content-type? %)
  :exists? (fn [context]
       (when-let [data (find-daa)]
             {:data data}))
  :handle-ok #(handle-ok %)
  :post! #(handle-post! %)
  :post-redirect? (fn [context] {:location (url-in-context "someurl")}))

As you might guess, this method known-content-type? is probably applicable for most resources. But how would you handle the exception to the exception? This is actually quite easy, as it turns out. In line with most examples of multi-methods I’ve seen so far, we’ve used a simple value to dispatch on. But of course a map can be a value, too. Given the need to override (specialize) the method for some resource, the idea is to define the dispatch method to return a map with the request-method and the resource. We then define the methods with appropriate values. The nice thing about this is that it’s very easy to arrange for default behavior for a request method by just leaving out the resource key — the dispatch function takes precautions not to add a superfluous :resource key in case none is added to the context by the resource.

(defmulti known-content-type?
  "Determine known content types depending on request-method"
  (fn [context]
(logging/info (str "Found resource: " (:resourceclass context)))
(logging/info (str "Method: " (get-in context [:request :request-method])))
(let [dispatchval {:request-method (get-in context [:request :request-method])}]
  (if-let [resource (:resourceclass context)]
    (assoc dispatchval :resourceclass resource)
    dispatchval))))
(defmethod known-content-type? {:request-method :post} [context]
  "Allow only application/json for POST requests"
  (logging/info "Determining known content-type for :post!")
  (when-let [content-type (get-in context [:request :content-type])]
(condp = content-type
  "application/json" true
  false)))      
(defmethod known-content-type? :default [_]
  (logging/info "Determining known content-type for :default!") 
  true)

(defresource some-resource
  :available-media-types ["text/html" "application/json"]
  :method-allowed? (request-method-in :get :post)
  :known-content-type? #(known-content-type? %)
  :exists? (fn [context]
       (when-let [data (find-daa)]
             {:data data}))
  :handle-ok #(handle-ok %)
  :post! #(handle-post! %)
  :post-redirect? (fn [context] {:location (url-in-context "someurl")}))


(defresource special-resource
  :available-media-types ["text/html" "application/json"]
  :method-allowed? (request-method-in :get :post)
  :known-content-type? #(known-content-type? (assoc % :resource special-resource))
  :exists? (fn [context]
     (when-let [data (find-special-data)]
       {:data data}))
  :handle-ok #(special-handle-ok %)
  :post! #(special-post! %)
  :post-redirect? (fn [context] {:location (url-in-context "specials")}))


(defmethod known-content-type? {:request-method :post :resource special-resource} [context]
  (logging/info "Determining known content-type for :post and special-resources!")
  (when-let [content-type (get-in context [:request :content-type])]
(condp = content-type
  "application/json" true
  "application/x-www-form-urlencoded" true
  false)))

With these definitions in place, the default for POST requests using this known-content-type? method would be to accept only application/json. However, the special-resource “overrides” this behavior to also accept regular form data. Posting to the various resources will produce output like the following:

2014-04-14 15:04:31,088 [main] INFO  utils - Found resource: liberator.core$resource$fn__3268@688dbd21
2014-04-14 15:04:31,088 [main] INFO  utils - Method: :get
2014-04-14 15:04:31,089 [main] INFO  utils - Known content-type for :default!
2014-04-14 15:27:14,974 [main] INFO  utils - Found resource: liberator.core$resource$fn__3268@688dbd21
2014-04-14 15:27:14,974 [main] INFO  utils - Method: :post
2014-04-14 15:27:14,975 [main] INFO  utils - Known content-type for :post!
2014-04-14 15:04:31,127 [main] INFO  utils - Found resource: liberator.core$resource$fn__3268@688dbd21
2014-04-14 15:04:31,127 [main] INFO  utils - Method: :post
2014-04-14 15:04:31,127 [main] INFO  utils - Known content-type for :post and special-resources!

Please note that known-content-type? has to be a known symbol (defined or at least declared) prior to be usable in the resource definition, whereas adding the more specialized method requires the special-resource to be defined — declaring it won’t be enough.

Using maps as dispatch values seems to be a nice and powerful tool to know about. There are, however, still some points where I see room for improvement:

• We would probably like to use the same mechanism for a ton of functions, always highly similar. E.g. the methods for handle-not-found and known-content-type? look highly similar on the structural level. Also, when you have multiple resources, the dispatch function for one method type (i.e. something like handle-ok) are probably always the same, so are the dispatch arguments (i.e. the media types our web application will handle). Maybe a macro would be useful here, but I haven’t thought it through yet.
• Handling the Accept header is actually way more complicated. Fortunately, liberator takes care already of choosing the “right” media-type (cf. again the liberator tutorial on content negotiation. However, as also discussed in the same section of said tutorial, there are more negotiable parameters which might come into play, e.g. language or encoding. This quite obviously could lead to some combinatorial explosion. While the approach using a map outlined above is a way to handle it, this approach is essentially mimicking CLOS’ dispatch on multiple arguments via a single argument dispatch.
• I haven’t even started to think about how one would approach the more advanced problem that Christophe is solving by using his MOP trickery generalized-specializers.
• The name of the method users-handle-ok isn’t really telling. Of course, a name like users2json or serve-users-view seem better suited to describe what the respective methods are doing, but this obviously would defeat the idea of using multi-methods and the associated benefits. Still, the name should probably not be tied so close to the resource definition. Using the name parameter of methods is one way to remedy this particular issue.
• Finally, apparently the slots of a liberator resource expect function objects. Liberator won’t just take the name of a function and do the right thing, it’ll throw an exception. Not a big deal, given that we might need to mangle the implicit argument (the context) anyway, cf. the :known-content-type?slot of the special-resource.

I recently did some coaching on how agile teams work, what roles there are and what are their respective responsibilities. It’s probably no surprise that one of the main topics was self-organization and what this implies for how agile teams work, also how to get there and how to solve the various puzzles (like distributed teams) occurring when working with bigger team setups. You’ll find the slides below. I was asked to talk about these issues because the team was having trouble understanding what was expected of them and how to organize themselves. In my experience, that’s a quite common situation to run into when the team members (or at least some of them) have no or only little experience working in an agile setup. But even when you already have some experience, it might happen that the team or at least important members of it lose the understanding of how to work together. And of course, that’s much more likely to occur when you interrupt the team working together, e.g. by putting some people onto different jobs, even if only for a short time. I guess this is what people outside of a team (i.e. managers) often forget: teams are usally only able to perform great over a longer time if they can focus on “it”, i.e. working on their tasks and working on their understanding of how to work together as a team.

Anyway, I started by thinking about why people are always so eager to talk about roles and responsibilities. I came to the conclusion that it’s mainly a matter of safety or lack of it. That’s why you have these discussions mainly at the beginning of a collaboration. The important point that became apparent to me is that roles and responsibilities don’t matter in a performing team (performing in the sense of Tuckman’s model). And the corollary to that is: When you don’t know what is expected of you (as a team member), your team is really just a group of people and you don’t know how to work together as a team (yet). Let me elaborate this: I believe for a group of people to work together successfully it’s most vital to have a common understanding of what needs to be done and to just do it (as the old sport company’s saying goes). Of course, you will need somebody who will just do it, but does this imply that you have to have a formal org-chart upfront that tells you who has which role? What if you happen upon things that need to be done that you did not think about when compiling your org-chart? You will need somebody to step up anyway, so why not just leave it at just do what needs to be done? Even better if all of your team have a common understanding on how to do things (cf. for example a commonly agreed upon definition of done).

My main (and very obvious) advice on solving the issue of “I don’t know what is expected of me” is to compile a list of things that need to be worked on and to get started. Whether this “list of things” is a backlog or just a todo list, whether it’s on a physical board, in an Excel file or in some other system, is not as important as actually working with this list: re-visit this compiled list every so often to update it, prioritize items, move things around, augment and transform it as needed. And of course, remove things to do from the list that are actually done. But do this in a way that it is clearly visible all the time to all team members. If you modify this list openly so that everyone can participate, everybody will learn about what is going on, you can have discussions that will help team members understand why something needs to be done first and which things are problematic. And, last but not least, people will communicate over helping each other out, give input on difficult questions and work together.

The other important ingredient for me is giving and receiving feedback. If you’re working in a Scrum-like fashion, review and retrospective meetings at the end of an iteration are the ideal places for this. I believe that the awkwardness that in particular the retrospectives have for people new to agile is an indicator of a company culture, team setup or personal attitude that does not honour open discussions and feedback. Retrospectives make people feel uncomfortable especially if they are feeling insecure. However, without feedback it’s difficult to learn, so organizations should in my opinion strive to provide safe environments for their people. A simple example of this is to have the retrospective facilitator use the “setting the stage phase” to ensure that the people feel safe (cf. this article on creating safe environments for retrospectives), maybe by having the team come up with rules for safe communication on their own or by simply reminding them of already existing rules. However, the important point for me is that team members should learn to give and take feedback in any situation. No, I don’t mean that it’s appropriate to interrupt your colleague every five minutes to complain about some weird code or that you should accept getting interrupted every so often. But, e.g., if Maria believes there are many issues with the code that Peter checks in everyday, it should be okay for both of them when she arranges a discussion between her and Peter on how to solve the problem. If Maria can provide her feedback in a way that is both personal and helpful, this will help building trust which leads to a feeling of safety wrt. working with each other. Typically, such discussions in between (as well as during retrospectives) will address many of the fears, uncertainty and doubt that trouble people about what is expected of them.

I believe that discussions about how a team should organize itself and surrounding tools for this (e.g. the RACI model) are mainly waste. Remember, during the time you’re discussing your organization, you don’t add anything of value for your clients. I would rather spend the time on making progress on the tasks that are waiting for attention and figure out how we work together while actually accomplishing something.