On Being Economical with the Truth

A while back, I attended a presentation about a suite of software that allowed IT managers to track in some detail the progress of a development project, via static code analysis, a range of source control metrics and other magic. Bathed in these insights, management could instantly track the detail of what was going on and draw conclusions from it.

Having finished, the presenter stepped back from his detailed graphs in anticipation, perhaps expecting uncritical plaudits from the developers in the audience. We clapped politely, and shuffled uncomfortably. Nothing in particular emerged from the discussions beyond some vaguely-expressed reservations. Some things are best are left unsaid, but we all appreciated, instantly, that such detail was often dangerous in the wrong hands.

Why would a development team ever want to conceal anything from their management? Well, all organizations flourish by being, err…selective with information. Every department carefully manages information about what they are doing, so as to prevent misunderstandings. If the entire truth is told at a meeting of the board of directors, there is always a gasp of astonishment and mutterings of discontent. The same holds true of a development team.

What actually should happen in the course of creating an application is messier and more complicated than the perceived wisdom implies. The art of developing effective software generally relies on there being several phases. First, there is a wild, creative phase where anything goes, and there is a lot of architectural sketching, rapid development of prototypes and wireframes. Alternative technologies and algorithms must be assessed. There is a second phase where there is enough consensus that the team can work together to create a likely candidate application, which is close enough to the final product. At this stage, major U-turns can be considered, even if it frightens the project manager. The final stage is a disciplined and highly structured march towards release.

A healthy creative process can and should resemble chaos. I know of no metrics that can measure progress effectively during this phase. In fact, those disciplines that are essential to the final phase of a software project are complete anathema to the first phase. Issue-tracking, for example isn’t appropriate. Mock-ups and prototypes are supposed to have bugs. Developers have to be actively discouraged from too much attention to detail. Even source control can be overkill when an entire codebase is very likely to be, and should be, jettisoned at short notice. The way that you manage technical debt is quite different.

When we are asked by outsiders whether we invariably use TDD, source control, issue tracking, continuous delivery, Kanban, Scrum, quality metrics and so on, we nod assent, unblushingly, as though it were a simple question. The truth is more nuanced, and more difficult for the orderly and mechanistic mind of many in middle-management. It is best to manage what information gets outside the team, because truth is often complicated.

  • 5080 views

  • Rate
    [Total: 8    Average: 3.9/5]
  • Dave Wentzel

    Love your writings.

    It’s not just “outsiders”…more and more devs are beholden to these processes and don’t know how to function creatively. Devs are assigned to “experimentation” teams and their first response is “ok, where’s the kanban board.” Um…we are experimenting here. Or you need a really quick MVP/MMF and they go do TDD. Um…TDD is valuable, just not when I’m trying to get funding for a product that is vaporware. I need basic features and a UI that looks like it is feature-complete. I really just need the dev to tell me what buttons NOT to click during the demo. But if you say, “don’t waste your time with TDD *right now*”, well, the reaction is akin to restrained violence.

    • Mike Cattle

      I understand the need to get up and running quickly for a demo, but how often does that “quick and dirty” code become the base that the production code is built on, once the project is funded?

      • Peter Schott

        I remember we had some consultants come in to do a mock-up for us of what they could deliver, but _nothing_ was wired up to any sort of database and the code wasn’t really designed to be enhanced. It was purely a Proof of Concept designed to show that they understood what the business needed and could wire up the proper functionality in a design that worked well for the users. Pre-determined responses/screens would launch when they navigated through the app to show what everything would look like. If we hired them, we paid for that time and they would then work on wiring things up, tweaking, etc. If not, they took everything and had minimal charges, but we were not left with something we could extend. Code lived on their machines and was mostly mock-up screens.

        That said, in most scenarios, that “this will only be used here” code quickly turns into “the whole company will die without this”. 🙂

      • Dave Wentzel

        That’s a different problem, one of management. That fear, which is well-founded in many shops, should never be a reason to stifle the creative process to get a good demo fast.

        This “creative” phase is not for every developer.

    • buggyfunbunny

      One of my colleagues, many years ago, lived by a mantra: “never making a working prototype, else the Suits will ship it.”

    • Gina Taylor

      Hi Dave,

      Congratulations on winning this week’s commentary competition. Please could you send a message to editor@simple-talk.com from your preferred email address and we’ll shoot over your $50 Amazon gift card prize.

  • AndyDent

    I have a modest proposal – your blog post prompted me to write up some thoughts I’ve been slowly stewing for about 30 years, since my ex-RAF boss used to ask “can I shoot you if you’re wrong?” – every time I said I’d thought something through.

    I have a fondness for elaborate names, such as the “Principle of Least Astonishment”, so I would like to offer up for your amusement:

    The Ratio of Appropriate Consideration. aka “Fractal Finking”

    Simply put, it is:

    Count the things you *have* thought about / Count the things you *should* have thought about.

    The ratio comes from an iterative and sloppy process matched to the chaotic nature of early system creation.

    START
    Start with the denominator – make a simple list of the parties involved and any functional goals, or just really sloppily estimate an arbitrary number of them. This is rough – you can and will revise it many times.

    Then the numerator is simply a rough count of each one of those items you *have* thought about, “enough”.

    The things in the list that sums up the denominator will fluctuate a bit over time but mostly grow.

    Your Count of Considerations at the top will vary much more.

    ITERATE
    You can choose two ways, and even flip between them.

    That rough count of thinking about doesn’t have to be from integers. You could have a simple spreadsheet list of things to think about and have a percentage. At this stage, have you done “enough” thinking about something, or just 20% of the thinking you would like?

    The other way to iterate is to reset the count.

    When you move onto slightly more formal evaluation, reduce how much things have been considered to, say, 10% (start an another column in the spreadsheet).

    Then go back over the list and start adjusting the Consideration level for each thing.

    CHOOSING LEVEL OF DETAIL
    The important thing about this ratio is you can choose to be as sloppy or detailed as you want and the ratio works just as well. You can use one version just to track having thought about involved parties, or “personas” (if you use them).

    It is extremely “fractal” in nature – iterating at more detail as you feel appropriate.

  • callcopse

    I’ll defer to JK Rowling, via, Dumbledore.

    “The truth.” Dumbledore sighed. “It is a beautiful and terrible thing, and should therefore be treated with great caution.”

    I’m not entirely in agreement here that the chaos phase is always required. On many project inceptions we can work with prior art assembled in the department – little in the way of erratic implementation required. Still, where you are designing a new way of working, or working with significantly new technology, then you do get that kind of craziness. I don’t think we could afford to always be doing that though – it would make life really hard and potentially unprofitable. It is however great fun!

  • CujoDeSoque

    Sometimes people really don’t want to know and forcing unwanted knowledge makes it worse as it cannot be denied and can’t be taken back easily. There’s a reason executive summaries, prefaces and high level overviews are part of a process.

    If I ask for a test machine and a copy of a server’s application environment, that should be all that an upper damagement needs to know. You have an idea, you are moving it away from the collateral damage zone and it will eventually have a result. How it’s accomplished is a matter of peer review in most cases.

    • S Charlesworth

      ‘upper damagement’. That’s a new term to me, love it!

  • TodConover

    I’ve been a programmer for 45 year. Three times, when asked by my boss’s boss how it was going, I told the truth, and three times I’ve gotten in serious trouble. Most people in this industry don’t know what they are doing; hence the fudge “Factor.” If we did, then all this wouldn’t be necessary. This should be an exercise in engineering, not an art form. If we are engineers, than there is no confusion at any stage of a project. You don’t have this kind of confusion, and accompanying falsehoods, when designing and building an airplane, why is a software project different? For whatever reasons, we’ve forgotten what works. Remember data modeling? That’s where you discover and document the requirements. It’s not hard, its not confusing; if you know what your are doing it is easy. Once you know what is required then you just build it, and you can provide accurate status. The argument against this is that you never know for sure what the customer needs. That’s very popular these days, but it is wrong. Go back to chapter one of data modeling and reread it carefully this time.

  • rogerthat

    Rather risky subject, great content, excellent comments by all.
    So at worst, our programming team is a bunch of hooligans until time demands measurable output.
    At best, our programming team has mastered the fine art of software development.
    The problem is systemic with what we are asked to do on a regular basis: do something different. Business never wants the same thing created time and time again. Yes, we can put some measures in place, some processes that assist with the how we do development, but all those critical trial steps leading to the code that is worth source control — they are unique every time.
    I never could stomach adding “engineer” to my programming title — an engineer is able to utilize hundreds, if not thousands, of learned, tested materials and techniques. I don’t think software development has the tangible goods to wear “engineer”.
    And don’t forget that much of the time our building materials are changing on us. Specific features of the components or source become stale within a few years. How many times has Microsoft rearranged the letters ADO for data access?
    I like agile development because it adds transparency between business and IT. Nonetheless, some discussions need to be within a developer-only meeting. Just like some business meetings need business-only.
    I think it all boils down to trust. You either trust your car mechanic or you don’t. If you do, you are going to take her advice when she says it would be better to spend $500 now vs. $1750 later.

  • Rajesh Kumar

    As Nick Fury in the Film, Captain America: The Winter Soldier(2014) says, “It’s called compartmentalization. Nobody spills the secrets, because nobody knows them all.”
    http://www.imdb.com/title/tt1843866/quotes?item=qt2102242

    This elastic force between the Management trying to know whats happening in the software project and the software developers keeping their cool and compartmentalizing the truth and shares it on the need to know basis, is very much essential for everyone to successfully alleviate stress and get the software bits out for customers and continuously evolve towards betterment.