How and Why Many Software Development Shops Fail
What I see a lot of, is organisations hiring code monkeys rather than professionals. Either they hire:
- the cheapest talent they can get their hands on. Now they want the best, but how much they have to pay the developer is the most important factor to them.
or
- the person that completes feature implementations as fast as possible (sometimes known or thought of as rock stars). Often young developers without a large amount of experience which causes the more Professional Developers to slow down a bit and think tasks through a little more.
Now, both approaches are short sighted. They hire code monkeys rather than professionals. Code monkeys write code fast and incur technical debt that is hidden at first, but over time slows the Development Team down until it can barely move.
The scenario
Code Monkey finishes his task much faster than Professional Developer.
Code Monkey is solely focused on completing his task as fast as possible. He cuts some code and declares the task done. Professional Developer thinks the problem through, does a little research to satisfy himself that his proposed approach is in fact the most appropriate approach for the problem. He organises a test condition workshop which solidifies requirements and drives out design defects via active stake holder participation. He drives his low level design with TDD. Makes sure he follows the coding standards, thus making future maintenance to his code easier, as it’s much easier to read. Asks for a pair to review his code or perhaps requests a fellow team member to sit with him and pair programme for a bit on some complex areas of the code base. Makes sure his code is being run in the continuous integration suite, that his acceptance tests (which have been driving his feature) are passing and the (security) regression tests are not regressing. Checks that his work complies with the Definition of Done. You do have a Definition of Done right?
Now what the Product Owner or software development manager often fails to understand is that it’s the slow (Professional) developer that is creating code that can be maintained and extended at a sustainable pace. Professional Developer is investing time and effort into creating a better quality of code than the developer (Code Monkey) that appears to be producing code faster. The Product Owner and/or manager don’t necessarily see this, in which case Code Monkey clearly looks to be the superior developer right (the rock star)? What also often happens is that Code Monkey rides on Professional Developers quality and adds his spaghetti code on top, thus making Code Monkey look like a god.
The Product Owner sees output immediately by Code Monkey that “appears” to be working very fast. He doesn’t see the quality being created by Professional Developer that “appears” to be working slower.
Time goes by. Sprint Review roles around. The stake holders love the new features that have been implemented and now want some additions and refinements. They ask the Product Owner to add some more User Stories into the Product Backlog. The Development Team pull these stories into a Sprint and start work. New functionality is added on top of the code that Professional Developer wrote previously. New functionality is added on top of the code that Code Monkey wrote previously.
Sprint Review roles around again and the stake holders are happy with the new features that have been added on top of Professional Developers code. Of course they have no idea that the underlying code was crated by Professional Developer. Now the stake holders have been using the software that had the new features added on top of Code Monkeys spaghetti code and they are starting to notice other areas of the application that are no longer behaving the way they are supposed to. This continues to happen and the stake holders are oblivious to the fact that it’s due to the code that Code Monkey is writing. They still think he’s a rock star because he appears to pump out code so fast.
So… while Professional Developer seems to be slowing The Team down and clearly Code Monkey is simply amazing because he delivers his features so much faster. The actual truth is exactly the opposite. Professional Developer is creating SOLID code and running at a pace that’s sustainable (a key principle of the agile manifesto).
The code that Professional Developer wrote is easier to modify and extend as it’s design is superior, due to being well thought out and driven by tests. His code satisfies the acceptance tests, so everyone knows it meets the living specification. It’s faster to add changes to his code because it’s easier to read and there are less surprises. If any other team member changes Professional Developers code which makes it no longer conform to the specification, the Accpetance Tests around his code fail, thus providing instant feedback to the developer making the change.
It’s the practices of Professional Developer that:
- provide the entire Development Team assurity that the software satisfies the requirements of the specification at all times.
- allow The Development Team to run at a sustainable pace.
- provide confidence in ongoing future estimations due to less surprises.
- produce code that everyone wants to work with.
- produce less error prone software that does what it says it will do on the box.
So… next time you as a Product Owner, Manager, or person responsible for hiring, is looking for talent, be very careful what you’re measuring. Don’t favour speed over excellent attitude. I created some ideas on what to look for in a Professional Developer here.
Scrum Teams can Fail Too
Velocity of the Development Team starts high then declines. Often it’s hard for people (including the Product Owner) to pin-point why this is happening. The Scrum Team may have started out delivering at a consistently high cadence. They appeared to be really on fire.
The code base is small but growing fast. As it starts to get larger, the Development Team starts to feel the weight of a lot of code that’s been hacked together in a rush. This causes the teams ability to release software fast to wane. A Scrum Team can get to this point quite fast, as they are a high performance team. When you get to this point, almost every change to the code base is hard. Make one change and something else fails. Routines are hundreds of lines long. Developers have to understand hundreds of lines of code in order to make a small change. Names are not as meaningful as they should be. Routines have multiple levels of abstraction, so multiple levels of code need to be understood to make a single change. Inheritance is over used, thus creating unnecessarily tight coupling. There are many aspects of the code that have become terrible to work with.
How does this happen?
How does the Product Owner know that the quality of code being created is not good? The Product Owner isn’t generally a developer so doesn’t know and even if he is, he’s not in the code day in, day out. Also it’s not generally the most important concern of the Product Owner, rather getting new functionality out the door is, so this is what the Development Team are rewarded for. When they pass a Sprint, the Product Owner is happy and praises the Development Team. The Product Owner has no idea that the quality of code is not as good as it needs to be to sustain a code base that is easy to extend.
So the Development Team does what ever it needs right now, to make sure they deliver right now (the current Sprint). Quality becomes secondary, because no one is rewarding them for it. This is a lack of professionalism on the Development Teams part. Bear in mind though, that each developer is competing against every other to appear as though they have produced the most. After all, that’s what they get rewarded for. Often what this means is they are working too fast and not thinking enough about what they are doing, thus the quality of the code-base is deteriorating, like in the example above with Code Monkey.
I’ve personally seen this on close to 100% of all non Scrum projects I’ve been involved with. Scrum Teams are sometimes better off because they have other practices in place that ensure quality remains high, but these practices are not prescribed by Scrum.
So… What do we do?
We not only reward the Development Team for delivering features fast but we also reward them for the sort of practices that Professional Developer (from our example above) performs.
How do we do this?
We add the practices that Professional Developer from above performs to our Definition of Done.
The Product Owner runs through the Acceptance Criteria which should be included on every Product Backlog item (preferably during the Sprint) indicating acceptance. Also running through the Definition of Done… querying the Development Team that each point has in fact been done for the Backlog item in question. This of course should be done by the developers themselves first. This provides the Team with confidence that the Sprint Backlog item is actually complete. Essentially, the work is not Done, until all the Acceptance Criteria points and Definition of Done points are checked off. This way the Development Team is being rewarded for delivering fast and also delivering high quality features that do what the stake holders expect them to do. No nasty surprises.
On top of what our solo Professional Developer did above, we should:
- Measure test speed and reward fast running tests
- Measure cyclomatic Complexity
- Run static code analysis and use productivity enhancing tools. This is not cheating, it’s allowing the developers to work faster and create cleaner code. This can even be set-up as pre-commit hooks etc on source control.
- Code reviews need to be based on the coding standards and guidelines.
- Encourage developers to commit regularly, thus their code is being run against the entire test suite, providing confidence that their code plays nicely with everyone else’s code. Commit frequency can be measured.
- The Development team should shame developers when they break the CI build. Report on how long builds stay broken for and shame when the duration is longer than an agreed on time.
- Most of these practices can be added to the Definition of Done, this way Developers can and should be rewarded for doing these activities. Even better, you can automate most of these practices.
Tags: operational efficiencies, performance, Scrum
Binarymist 
February 1, 2014 at 19:55 |
Kim,
Great article! You mention PO Acceptance at the Sprint Review. Could the PO also do acceptance in the Sprint itself, as soon as the PBI is completed?
February 1, 2014 at 23:31 |
Good comment! Updated post to reflect your thoughts, as yes the PO should be involved in the process during the Sprint so there are no surprises come Sprint Review.
February 2, 2014 at 20:09 |
Interesting, and lots of good stuff! I have couple of questions though, I hope you don’t mind?
“They [the stakeholders] still think he’s a rock star because he appears to pump out code so fast.” Why do they know this? Why does individuals pick/take on/get features assigned to them?
“… querying the Development Team that each point has in fact been done for the Backlog item in question”
The PO asking these questions, I see as a ‘late in time’ attempt to check for quality adherence. In Scrum (and Lean/Toyota Production System), we build quality into the process, not as a last step. We start with automating acceptance tests, instead of at the end asking QA (or PO) to verify quality of the system after it’s been produced.
Along the same lines, I’m not a fan of blame (and shame) game. If there’s a broken build, there should be no 1 person that takes responsibility, but at least 2 – a pair. And again, do root cause analysis to identify why our process gave this result, then try to fix the process – not introduce ‘too late (in the CI server) quality checks’. And the team should care about this, no-one else (no blame). No other team should see the broken build, nor get any half-baked artifacts from it.
In general, the Scrum Development Teams you describe at first – as being actual Scrum Development Teams – seems to have missed the point of them being responsible for, and taking ownership of, quality. This is the only thing they commit to: to follow the Definition of Done – to delivering (agreed upon) quality product or service. Scope is flexible, quality is not.
(I’d like to see more pair programming in your article. Less individuals, more pair, more teams.)
I’d love to see you in one of my Professional Scrum Developer classes, I’m sure you’d bring great experience and thoughts into the room. I also think that the light you see at the end of the tunnel, would be even brighter and clearer. 🙂
Keep it up!
February 3, 2014 at 22:58 |
Not at all.
“They [the stakeholders] still think he’s a rock star because he appears to pump out code so fast.” Why do they know this? Why does individuals pick/take on/get features assigned to them?”
Not sure I fully understand your question? When a developer finishes a task they start another one. Sometimes it’s more efficient for a developer to knock off a task by themselves.
“The PO asking these questions, I see as a ‘late in time’ attempt to check for quality adherence.”
Indeed. Although the Scrum Guide clearly states: “The Product Owner tracks this total work remaining at least every Sprint Review.”
As I’ve mentioned, this is done by the developers themselves first.
“In Scrum (and Lean/Toyota Production System), we build quality into the process, not as a last step.”
I’m a little confused here, I’m guessing you didn’t read the entire post? This is exactly what https://blog.binarymist.net/2014/01/25/essentials-for-creating-and-maintaining-a-high-performance-development-team/#professionalDeveloper addresses.
“We start with automating acceptance tests, instead of at the end asking QA (or PO) to verify quality of the system after it’s been produced.”
Again, I think you missed the links I provided on this: http://www.slideshare.net/kimcarter75098/moving-to-tdd-bdd
If driving your development with tests https://blog.binarymist.net/2012/12/01/moving-to-tdd/ and all the other practices I’ve listed here are not “building quality into the process” I’d be keen to understand what you think is?
“Along the same lines, I’m not a fan of blame (and shame) game. If there’s a broken build, there should be no 1 person that takes responsibility, but at least 2 – a pair.”
I’d have to disagree on this one. Just as the PO is the single wringable neck, the developer that breaks the build should be held responsible.
“In general, the Scrum Development Teams you describe at first – as being actual Scrum Development Teams – seems to have missed the point of them being responsible for, and taking ownership of, quality. This is the only thing they commit to: to follow the Definition of Done – to delivering (agreed upon) quality product or service. Scope is flexible, quality is not.”
Again it seems you’ve completely missed the point? That’s exactly what the post is about. Scrum is very quiet about quality (mentioned 3 times in the Scrum Guide and 0 times in the Agile Manifesto). Adding quality focus into the mix where Scrum doesn’t detail how to obtain it.
“(I’d like to see more pair programming in your article. Less individuals, more pair, more teams.)”
It seems you missed the links I included in the post with references to pairing? Here they are again:
http://www.codinghorror.com/blog/2007/11/pair-programming-vs-code-reviews.html
https://blog.binarymist.net/2012/03/24/how-to-optimise-your-testing-effort/#pairReview
https://blog.binarymist.net/2013/03/02/how-to-increase-software-developer-productivity/#codingStandardsAndGuidelines
The last link clearly shows in Scott Amblers Cost of change curve that in-deed Pair Programming is “the” biggest money saver when it comes to driving out defects at the earliest possible point in time.
February 11, 2014 at 12:33
Kim,
I did read the entire article, but didn’t follow a single link. I thought (and still think) the good ideas in this article stands on its own. I assumed links to other articles simply supported your case, not that you’d have to crawl those out to understand the point your making.
I’ll try to rephrase and be a little more verbose – to me it looks like we’ve misunderstood each other. 🙂
You’re right – Scrum says nothing explicit about quality, except that it’s the Dev Team’s responsibility. That’s the whole reason of the team pulling work in, to allow them to not have to bend on quality.
The DoD defines that level of quality. It’s the documented, shared understanding of what the team has accomplished/finished when they say “Done”. (I personally push for the flip side too, that the team is equally transparent about what it will NOT have done – typically because the Dev Team lacks resources (licenses etc) or skills. This list of items is something the Scrum Team needs to work to resolve.) The transparency this brings makes it less problematic for the PO to understand where the product’s at, it’s readiness with regards to being fit for purpose and to what extent it’s shippable/releasable.
When a build is broken, it’s the team’s ways of working that “allowed” this to happen. The whole team should be interested in fixing the process (TPS thinking, change the process to build a process that results in high quality), not just one individual. If you have code review, and the code was reviewed – then at least two persons “granted” that code being moved on down the “production line”. If not, well perhaps they, as a team, should consider start reviewing before pushing code further down. They might argue that they don’t need code reviews, which can be perfectly fine. As long as the team responds and collectively takes responsibility for quality, then I’m fine with it.
And, it’s OK to disagree. I would like to understand why you believe/think it’s better if one individual should be held responsible.
While PO may track progress work of towards a goal (this is on Product Backlog level – PBI level), it’s not the job of the PO to check that the DoD has been followed. This is the responsibility of the Dev Team. From agile manifesto’s principle: “… motivated individuals. Give them the environment and support they need, and trust them to get the job done.”
I’m only objecting here because I feel like we’re back to early school years where a student hands in a note from his/her parent(s) saying “Yes, little Bobby Tables has been to the dentist” for inspection by the teacher, to verify that little Bobby wasn’t skipping class. With grown up people, we trust them instead. PO should trust the team to follow the DoD.
So, I don’t think you need to add explicit checks for quality. The DoD is there before starting the Sprint, and understood by both PO and Dev Team. I hope this cleared things up. 🙂
February 3, 2014 at 06:17 |
I like Fredrik’s comments, and I tend to agree on more pairing. Even having to do this “PO asks about the DoD” process speaks to a lack of understanding on the team around the DoD process, and/or a lack of respect for it (maybe the code monkey is a maverick and thinks he’s too good for a quality standard). I would work hard on make sure everyone is educated on why the DoD exists. I would also work hard to provide DoD transparency *eariler* in the process than PO acceptance… If you’re not into paired programming, then at least paired code reviews — and have the developers keep each other honest about the DoD.
Having said all of the above, I don’t have a problem with the creative and innovative way that the author has attempted to solve the problem. I think it’s a great improvement. He is educating the PO and Dev Team on the importance of the DoD, he is coaching it well in such a way as to have the “rest of the team” own the implementation, and this allows him to not appear as the “Scrum cop”.
A similar practice I have used in the past is to have the DoD checklist be in the “Done” column of the Scrum Board.. and I encourage the team to always run through that check list before they move that story to Done. (possibly just after a Daily Scrum, so it’s done as a team, etc).
Whatever education and transparency we can bring to DoD is a great thing.
I’ll also give big props to Fredrik — I have observed him personally and he is an excellent trainer — one of the best I’ve seen. His knowledge of Scrum/Agile and very deep technical knowledge are outstanding. Those who know me know that I have very high standards, so I don’t offer this endorsement lightly. If you get a chance, go to one of his classes!
February 11, 2014 at 19:48 |
Fredrik
Because the process has allowed one developer to break some code, doesn’t necessarily mean the process is at fault. The developer may not have run tests locally if there are tests. If there are insufficient tests then it may be the process. Ultimately I think it comes down to the developer writing the code. As you say, developers are adults and we should trust them. If they are code monkeys they will break the build.
“While PO may track progress work of towards a goal (this is on Product Backlog level – PBI level)”
The Scrum guide says “A Sprint Review is held at the end of the Sprint to inspect the Increment”. The Increment is what comes out of the Sprint (this is on Scrum Backlog level, not PBI level). This means the work that comes out of the Sprint is inspected by the PO at Sprint Review. I think it’s pretty clear.
February 12, 2014 at 10:29 |
I’m going to agree with you both here. I like the PO asking about the DoD(in the context described), but I would like it *better* if the team held each other accountable to the DoD, and the PO never had to ask… but of course the PO should always have at least a lay understanding of the DoD.
> This means the work that comes out of the Sprint is inspected by the PO at Sprint Review
Kim, what gave you this idea? I mean, the Scrum Guide does not specify that the SR is for the PO — it says it’s for the Scrum Team and key stakeholders to inspect the increment.
February 12, 2014 at 13:50
The team does hold each other accountable to the DoD. I thought that was obvious sorry. Scrum Guide: “Development Team of the Scrum Team must
define a definition of “done” appropriate for the product.” I guess this could be construed as meaning the dev team define the DoD for the PO. It’s good to clarify this as you have done I think Charles.
In reply to your last paragraph. Agreed. Although the Scrum Team has been inspecting the increment all Sprint. What’s the point of inspecting it again (redundant work). I see this as the PO and the stake holders inspecting the increment more so than the Dev Team, as they have been more intimately involved in creating the increment than anyone else.
February 12, 2014 at 01:12 |
I disagree – if the process didn’t catch the breaking of the code, then the process may be modified to catch a similar breakage in the future. Improvement? That’s debatable. If you see the work the team carries out as the implementation of the process, then I think a broken build is an outcome of the process.
Not sure we’re closing in on this one. 🙂
I partly agree: the Increment is inspected at the Sprint Review, and Done PBIs may be demonstrated. This is all Product Backlog level in my view. The detail of how the work is carried out during the Sprint – after Sprint Planning and before Sprint Review – is a concern of the Development Team. The PO may obviously be interested, but it’s the Development Team that self-organizes around turning the PBIs into a Potentially Releasable Increment.
“A Sprint Review is held at the end of the Sprint to inspect the Increment and adapt the Product Backlog if needed. During the Sprint Review, the Scrum Team and stakeholders collaborate about what was done in the Sprint. Based on that and any changes to the Product Backlog during the Sprint, attendees collaborate on the next things that could be done to optimize value. This is an informal meeting, not a status meeting, and the presentation of the Increment is intended to elicit feedback and foster collaboration.”
It’s not a quality check gate. The Increment is not inspected by the PO “alone” and “accept” the delivered Increment – it’s a collaborative work around what was Done (and not), taking every opportunity to maximize the value gained from future investment (next Sprint). In order to understand what to do next, we need to look at where we are now, so we inspect the Increment. Not to assess quality of the outcome, but to assess what to do next in order to achieve goals (business value).
February 27, 2014 at 20:48 |
Thanks! That was outstanding 🙂