Monday, August 05 2013
Keywords: Technical debt, technical bankruptcy, software project management, Agile
Keep Ignoring those Technical Debts and you Will Certainly Hit Technical Bankruptcy
At a wedding two days ago, I got to converse with an old colleague of mine...a software architect. When I asked him how he was doing at work...his face saddened and he sighed...he took a sip of red wine and started to tell me a grim tale...about a group of 8 good guys having worked hard for 2 years on a project...that unfortunately never saw the sun light.
The following is a real story of a technical debt whose high amount of accrued interest eventually bankrupted a once promising enterprise caliber application being crafted by 8 smart guys.
Whether you are in the middle of a software development project, or releasing a product next week, or at the conceptual phase of an exciting project…I invite you to read this story and to learn from others’ big mistakes. May your enterprise caliber solution be saved!
Without further ado…here goes my friend’s story:
“The group was comprised of 10 people: 1 project manager, 1 SCRUM Master, 2 senior middle-tier developers, 1 senior database developer, 1 senior U.I. designer, 1 mid-level developer, 1 junior and 2 dedicated QA staff.
A year and nine months had passed since the start of the project...the middle-aged SCRUM Master had two decades of software development under his belt and he knew so well that the accumulated level of technical debt in the project had surpassed the MAX ALLOWABLE DEBT threshold. But he had chosen not to spare his energy fighting this...as he was fighting other battles at the company at the same time.
The senior database developer, with the same number of years of experience, had the same exact gut feeling…all his requests and pleas for stress testing & performance tuning had been graciously “postponed”…“deprioritized".
All other folks in the group were sporadically heard complaining about the fact that a huge percentage of the 2000+ unit tests had been systematically failing during the last several months…specially after a huge refactoring job was implemented 4 months prior...and that at least 4 weeks were needed to fix the failing unit tests…and until then the unit tests were not deemed as trustworthy or a good source of reference.
They were right.
Basically unit tests had become completely obsolete…so every time a new feature was added, or some refactoring was applied…no unit tests were referred to by anyone to validate the changes!!! In fact in the last 3 months leading to the Beta release of the software at least 100 different small to large refactoring changes were applied to the source code at every possible tier…all without the help or validation of any unit tests!!!
The junior and mid-level developers in the team knew something was wrong…they were stressed out due to the high & steady flow of bugs coming up every day…but they didn't want to be the negative ones...the complainers...they kept hush...thinking that by doing so they were good employees and were on way to their huge salary increases and hefty bonuses.
And then came the Product Manager (PM) who absolutely loved her “little baby project”. She knew everything about that project inside out. In fact she was known to write emails off at 2 a.m. or 10 pm, any day of the week. Everyone in the group really liked her and they all got along very well...so in time an unspoken never-say-no-to-PM culture developed in that group...you see most devs liked her as a project manager...so no one wanted to give her a hard time... anyone who opposed her in any way would get a hard time to get the message through. These are real signs of problems in a group...when developers look after the interests of PMs and not after protecting and defending software development principles. One of these pillars being "lowering the technical debt".
The PM was rather hasty to get a release out…and that hastiness basically lead her to the dark side of accrued technical debt.
You see the PM had simply rejected every code clean-up requests…hadn’t approved a single week of unit test fixing during the 2 years! All “performance tuning” requests simply put on the back burner.
She loved to squeeze in as many new features as she could into every sprint. Nothing else mattered. She was always heard saying how proud she was of this project, and how many tools and features the application truly possessed…how useful it all was to the clients…she was 200% certain that dozens of big clients were going to beg to buy this software. And locked in that hope...all sorts of bells and whistles were added to the first release of this app. Bells and whistles that could have honestly waited till release 2 or 3 or even 4!!! The software could have been ready after 9 months and not take 2 years!!!
She was always heard saying that she had FULL TRUST in her team, which she considered as being all a bunch of experts. Seniors could not possibly make mistakes she thought. Senior staff would not create anything slow or dysfunctional she was heard saying. Seniors’ level of knowledge and ‘how to’ was so high, that anything produced by them had to be perfect...and if a problem was to be found one day...she was sure that the fix would be as quick as half a day... There were no worries…technically speaking.”
Yes a seasoned experienced developer can write superior code...that statement by itself is 100% true...However the reality is that every developer on the face of the earth also does write fast-messy code, only to be "cleaned-up" later. In time the accrued amount of technical debt in that group became stronger that the strength of all the seniors of the group altogether. There will always be one hay that will break the camel's back.
This software went to Beta…and never made it out alive… Real corporate clients had actually tested the beta software. Such high number of bugs were found that the software could hardly even be installed properly. For a few persistent clients that actually were able to install and configure that monster app, the pitiful low performance became a deal breaker.
just after the Beta release, the lead developer left the company. A month later the SCRUM Master was appointed to another project with higher priority...weeks later another senior developer in the group was "lent to another project for 2 years"...then the database developer quit for a better job. And to put salt on the wound they were all replaced by new staff that had to learn and understand that domain.
Two years of development went down the drain…along with many dreams of an eager PM. The VPs in the company decided to politely "shelf the project"...put it aside in the permanent back log...They silently killed it.
Cause of the problem: High levels of accrued Technical debt leading to total technical bankruptcy.
Usual causes of technical debt (and by addressing these points you could minimize the technical debt level):
- 1. Growing number of failing unit tests that are not proactively being addressed. Failing unit tests must get fixed in 2-3 days latest...within the same sprint ideally.
- 2. Not allocating time for performance tuning user stories. Procrastinating stress and performance tests.
- 3. Seniors leaving the team…new folks coming in have to learn the domain, to understand it…it takes months…and it is highly error prone. If possible make it possible for the new replacements to get in touch with the seniors who left for at least 6 months, to ask questions.
- 4. Middle-tier developers were not given the chance to clean up after their messy 1st try codes.
- 5. Same goes for database developers!
- 6. Same goes for U.I. developers!
- 7. The team had wasted too much time doing various “proof of concepts” user stories. A few is good and normal...too many is actually a bad thing.
- 8. A project with high incidence of open Software Vendor support tickets: per example too many Microsoft support tickets opened...is a good sign that the developers were left to their own devices to choose the technology with which to develop…and they obviously chose the hottest dev tool out there that had just came out night before…which is as always full of bugs and very risky…requiring lots of open support tickets…These support tickets end up taking too much time and resources and create uncertainty, complexity and overhead in the project. They usually add to the technical debt in several different ways.
- 9. Technical debts are normal part of development cycle…BUT NO ONE ADDRESSING THEM PRAGMATICALLY IS AN ACTUAL CAUSE OF THE BANKRUPTCY.
- 10. Seniors who put up with PMs/POs who just ask ask ask for more features. Seniors out there: stand up for our beliefs and get the message through: at least 4 weeks of performance tuning is required in any enterprise caliber application.
- 11. Too much legacy “old ugly stuff” code to clean up or to learn.
- 12. Changing the technologies in the middle of active sprints…per example going from Visual Studio 2010 to 2012, and from .net framework 3 to 4 in the middle of a busy sprint…just because a couple of seniors pushed for it…and no one dared say no.
- 13. Going from VS2012 to VS2013, and to .net 4.5 only 5 months after the above-mentioned change!
- 14. A team where all members do NOT have the same versions of Visual Studio, or SQL Server installed on their workstations…that leads to 1001 unthinkable issues and wastes of time. This point is especially important for SQL Server: all team members should install and use the lowest version of SQL Server that your app supports. Ignore this and you could end up having last minute gotcha’s.
- 15. Too many juniors in the team: they would give in to the pressure too easily…they’d write sub-optimal code…it’s all normal of course.
- 16. Too many seniors in the team: too much ego…that leads surprisingly to too much silo work as seniors try to not over-step each other...…too much change of technologies suggested as seniors love playing around with new hottest toys…juniors having to learn all the hard stuff that seniors are introducing into the project all the time...all this adds to technical debt directly or indirectly.
- 17. Bad or non-existent automated builds.
- 18. Flaws and bugs being camouflaged all the time as “It’s a feature” usually come back and hunt you at the end when the software is actually released into client’s hands. The client will not be fooled. They have the money and they will ask for what they think the correct behaviour should be.
- 19. Lack of IT support and technical infrastructure in a company contributes indirectly to technical debt.
- 20. Supporting more than one RDBMS type by your application dramatically adds to the speed of technical debt accrual. Example would be supporting Oracle as well as SQL Server both. I have extensive experience with this topic. I would strongly recommend against this. Go with one RDBMS...for a long list of reasons. This topic deserves its own article.
- 21. QA Staff becoming too friendly and buddy-buddy with developers actually leads to accrued technical debts...I have seen this happens many times...the testers closing an eye on bad coding habits and mediocre functionalities and failing unit-tests...because they are friends with the developers...they discuss work matter at lunch hours...so everything must be fine they think...They in a way let the debt go on and to accumulate by not opposing it.
- 22. Over complicating the projects...creating a gazillion services and queues, and then making more services to fix the limitations of the first group of services...and then some.
Complex application designs occur due to over-excited individuals (like kids in a candy store) who are given the time and chance to abuse the system, these folks are not too well scrutinized and monitored, so they go out of their way to use every single existing new technologies out there and turn the so called enterprise app's middle-tier into a far-from-simple-impossible-to-install-and-even-more-impossible-to-maintain complex set of code that could only serve to create dependency on the continuous employment of the very individuals who developed this over-complex labyrinth of logic in the first place.
Stick to simple designs for the first two releases of your application. And once your application has been shipped, tested, PURCHASED, deemed profitable, and actually has demonstrated certain potential of a bright future; then and only then you should start adding bells and whistles to it, time and budget allowing. Unless you want to waste 2 Million dollar of company's budget to be a Microsoft lab guinea pig, for the sake of learning the newest and hottest fads out there.
Need to take data out of OLTP and into a Data Mart so that it could be queried? The entire thing could be written by two developers for one DBMS in a simple way in X amount of time; or you could put an army of 5, 6, or 8 people on it for 20X amount of time...the army wouldn't want to finish the project in X/3 length of time ...so what do they do? They create unneeded and fake complexity. Fifty services, several MSMQ queues, 5 Microsoft support tickets later...the same objective of Merging Data and Querying the Data Mart is accomplished...with 1000 times the complexity...and it took 20X time longer. It is next to impossible to install this beast...and any future service pack of any DLL at any level will simply break it in time to come.
Amount of Accumulated Debt: GYNORMOUS!
Written by Montreal DBA Team