2023-12-07T14:47:09+07:00

Doing It Well

December 7, 2023/Eli Mydlarz/Leave a comment

If you care about doing it well, you are most of the way there.

That’s no small thing. Doing it well means caring about improvement more than ego. It means being challenged instead of comfortable, humble instead of prideful, open instead of defensive, earnest instead of cynical, and hard-working instead of lazy. It is believing the best way is still out there and going on a mission to find it every day.

If you can find it in yourself, you are very lucky. If you can practice it well enough that others join you, then you can build a great team – and a great team is something very special.

Delivery Today and Tomorrow

May 3, 2022May 4, 2022/Eli Mydlarz/Leave a comment

My way of thinking about technical debt is a bit different to what I usually hear 😅 Let’s start by rejecting the usual framing – the idea that we are in some eternal struggle to find the right balance between speed and quality.

We’re not balancing speed and quality. We’re balancing delivery today against our ability to deliver tomorrow. We care a lot about delivery today, which may tempt us to compromise on delivery tomorrow. But do you really think that tomorrow, we will care less about delivery? Has that been your experience so far?

There are cases where delivery today is an existential necessity, but they are rare. Even startups need to deliver consistently on a long enough timeline for technical debt to catch them. But that doesn’t mean we should spend today developing complex architectures, implementing fancy design patterns or factoring new code exhaustively – there are good reasons to create technical debt.

Imagine you are on a trip and have just arrived at your hotel. Your luggage is full of useful items you may need during your stay. Do you assume you will use every item and unpack them all right away? Do you immediately fill cupboards and drawers with your belongings? Of course not! You start by keeping everything together so you know where it is (high cohesion) and unpacking things as you need them (YAGNI).

People give similar advice about building microservices. Don’t try to draw service boundaries at the start because you’ll get them wrong and it will be hard to fix. Keep things together in one service (high cohesion) and extract them only when boundaries become clear (YAGNI).

There are some life lessons here for us!

We are usually wrong at the start
Arranging is much easier than rearranging

That first point is important. When we start building something, we are probably wrong. Wrong about how our domain works, wrong about how to model it, wrong about which parts of the system will need to be extensible, wrong about our cross-functional requirements – there are so many things to be wrong about! The more heavily we factor our code, the more we pour concrete on our wrong assumptions. The more we arrange things at the start, the more difficult they will be to rearrange later when we find out we were wrong.

This “we’re probably wrong at the start” thing can be a bit tough for experienced people to accept, as we may have built our professional ego around the idea that we are good at being right. If you think you can get things right the first time, this post isn’t for you.

But if you’re with me so far, then my advice is this. Apply what we’ve learned about luggage and microservices right down to the class and function level. Don’t heavily factor your code at the start – just do enough. To build enough, we can use outside-in, test-driven development, test-driven design and close collaboration to produce the simplest, smallest, least-factored thing that works. Anything we are tempted to do beyond that, we can consider technical debt.

Working this way, our initial solution will be:

As simple as possible, because we will produce it outside-in while practicing test-driven development
Well structured, because test-driven design will guide us in writing data-driven code, separating concerns and avoiding complex conditional logic
Using domain language that makes sense to our business, because we are working closely with non-engineers
Sensible to our team, because we are mobbing or pairing with rotation
A step in the right direction, because we have thought about the problem holistically (but a small step only, because we have not thought too hard or too long)

The result of all this will be some software and some technical debt, so when and how should we pay that debt off? A few years ago, I would have suggested we briefly record technical debt on a technical debt wall, with axes for effort and value. I’d have suggested reviewing the wall at every story kick-off and picking up items that will make delivery of the coming story easier. That’s not a bad approach – it’s better than what most people are doing.

Now I believe technical debt walls are a local optima. In a team working as we have discussed here, categorising and prioritising technical debt is of little value:

Our opinion about how to improve something will change as our skills and knowledge grow
Our opinion about what is worth improving will change as we are asked to adjust and extend our software in ways we did not initially predict
We may not work in that part of the codebase again in a relevant timeframe anyway

It is more sensible to ask – before making any change to our software – “Can we improve our engineering design *a bit* to make the coming change easier?” Why only “a bit”? Because we don’t want to engage in a wholesale, big-bang redesign based on the need for a single change. Every time we change our software, we want to improve its design in a small increment, so that the kind of change we are trying to make is easier next time. This way our design will gradually evolve to support the kinds of changes we are actually making, in the areas we are actually making them. We don’t need to gamble on what should be extensible, how extensible it should be and in what ways we will need to extend it.

Put everything together and the entire approach is super simple:

Build the simplest thing we can
Every time we change it, make the change easier with a small improvement to our engineering design

With this approach in mind, the dichotomy between delivery and technical debt is exposed as false. We improve our engineering design when we deliver something and when we deliver something, we improve our engineering design. There is no need to categorise work as ”technical debt” or “delivery”. Such categorisation and the careful management that usually follows is a useful local optima for teams that are not evolving their engineering design continuously enough yet, but it shouldn’t be our ultimate goal.

“Technical debt” as a model for thinking about the need to continuously improve our engineering design is a good raft – it can help you get across the river as you learn how to do the right amount of work up-front and master improving it gradually over time. But when you get to the other side of that river, you should put the raft down.

Pressure is a Resource

November 18, 2021June 19, 2022/Eli Mydlarz/Leave a comment

We talk a lot about the best ways to work, but we don’t talk very much about how to move our organisations toward them.

As a former consultant, I grappled with this every day. Even with a clear idea of some alternative way of working, a large team of people ready to teach it and good executive support, it is still very difficult. One reason for this is that we hastily squander a powerful resource – pressure.

When a problem impacts our business, organisational pressure is created. We all want to relieve that pressure and that shared desire is a special thing – we can harness it to effect positive change. Unfortunately, many organisations react reflexively and take the path of least resistance. That is how pressure differences work, after all. We look for quick fixes like overworking “temporarily” or juggling more WIP. But quick fixes are not always good fixes.

What if we are only moving the impact somewhere less obvious? E.g. we have started a critical piece of work to placate stakeholders, but the resulting increase in WIP is causing inefficiencies across all work? What if we are only changing the timescale of the impact so that instead of being hurt a little every day, we get hurt a lot all at once? E.g. we are overworking somebody and it will blow up in our face when they quit.

Our quick fixes may actually leave us in a worse place than before. The root problem is still not solved and there is little pressure left to drive improvement. We wasted our chance.

It’s important that we not reflexively relieve pressure. Pressure helps us grow and change if we channel it wisely. We may need to use some of that pressure to make the immediate impacts of our problems manageable, but to build a great and enduring business we must direct as much of that pressure as we can into lasting solutions to our deepest problems. So spend your pressure wisely and don’t rush relieve it entirely. To improve continuously we need pressure and with no pressure, we can easily get stuck.

If you want a fleshed-out example, you can read my article about adopting trunk-based development. It’s an approach that offers some benefit at each stage, while still leaving or creating enough pressure to drive us toward the next one.

Focus on Technique

November 18, 2021/Eli Mydlarz/Leave a comment

If you want to be a better competitive swimmer, you don’t ask for feedback on your lap times and trophies – you focus on improving your technique and know that results will follow.

Most software developers are preoccupied with the properties of good code, but knowing what it looks like is not the same as knowing how to write it. A good critic isn’t automatically a good creator.

How does the length of your red-green-refactor cycle impact your results? Does working outside-in or inside-out change your solution? Do different pairing styles lead to different outcomes? Do you get simpler code by thinking through the problem in advance, or by being deliberately naive? Does writing failing tests instead of passing ones impact your implementation? Do your team members work through problems in different ways and if so, what are the advantages and disadvantages of their approaches?

If you can explain SOLID principles but you can’t answer these questions, forget about *what* for a second and start thinking about *how*.

Pair with your teammates. Let them show you their way of thinking and working and the next day, ask to show them yours. Set aside time together to reflect on your experience and discuss what you want to do differently next time. Do experiments together and share the results with your team.

Focus on technique.

How to Stop Branching

August 9, 2021August 10, 2021/Eli Mydlarz/2 Comments

I know several developers that were forced into trunk-based development without preparation and had a terrible experience. I don’t want to see that happen again, so I’m offering up this post as a practical guide to help your team safely climb down from their branches and practice continuous integration.

If you’re brand new to thinking critically about branching, you can read Martin Fowler’s explanation of continuous integration. If you’re a little further in, I love this blog post by Dave Farley. If you like branching and want to keep doing it, this isn’t the post for you.

Branching is a local optimum – it works just well enough for many teams to tolerate. It requires a number of interdependent practices and when we change one by itself, the system is degraded and the team reverts. When we change all of them together, we throw everything into chaos and nobody wants to try again. It’s a Gordian Knot I’ve spent my whole career untangling.

It has a way of recursively reinforcing itself. By facilitating working independently with limited collaboration, it delays feedback and causes rework during the pull request process, which in turn makes us feel that pull requests – and by extension branching – must be very important. Andrew Cain once told me that recursion is the most powerful force in the universe. Instead of fighting it, we’re going to take steps that will improve our lives now while preparing us for trunk-based development in the future.

Build in quality

Nobody wants to integrate poor quality changes into an unsafe codebase. That’s one of the things people attached to branching are trying to avoid – they want to know that each change is safe.

We can help them achieve this by driving the adoption of quality-focused development practices like:

Test-driven development
Overlapping coverage across multiple layers of testing
Tests as documentation
Static analysis
Linting

These techniques are more effective at catching defects than pull requests and also provide earlier feedback, reducing rework. They will add a lot of value, even if you keep branching.

Work in small steps

Even people who really like pull requests usually agree that smaller pull requests are preferable, because they tend to be reviewed faster and more thoroughly. More frequent integration also reduces the chance and size of merge conflicts.

We can switch from feature branching to task branching and work in a series of small, safe-to-deploy steps with a pull request for each one. We’ll spend less time debugging, receive earlier feedback from our peers and enjoy faster and more thorough reviews.

Many people are used to tearing everything apart and putting it back together again on their branch. It may take some people a lot of effort to learn a more incremental approach and you will have to be patient and persistent during this phase.

As you improve, you can begin working exclusively in small, production-ready commits. Even the handful of commits made on a small task branch can be made using the red-green-refactor cycle to ensure that your software is working at regular intervals, with only small changes between each known-good state. This will radically reduce cognitive load and debugging effort.

Use feature toggles

One challenge of breaking our work into small task branches is that we must decouple deployment and release. After all, we want to integrate our code – at which point it must be deployable – without releasing our unfinished story.

We can use feature toggles to achieve this – just think of it as runtime branching. We integrate our code, but delay releasing it by disabling it with a feature toggle.

In addition to facilitating more frequent integration, feature toggles allow us to quickly disable problematic features in an emergency. With the right tooling, we can even use them to do pre-release testing in production. Think of all the time we’ve spent trying to maintain production-like environments and test data when we could have just tested in production all along!

Feature toggles can be complex to manage. Putting them in the right places, tracking when they can be turned on and remembering to remove them may take some new tooling, some practice and even some workflow changes. Don’t rush this step – we need to be great at this before we can practice trunk-based development.

Collaborate more

Pull requests are now more frequent, but without increased collaboration this is usually slow and annoying. A little tension helps drive us to continue improving, but we have to guide our team carefully in this phase so they don’t learn the wrong lessons and go backwards.

Substantive pull request discussions cause unnecessary rework and can be taken as a sign that we did not collaborate well. We can eliminate this by working together throughout the development process using techniques like:

Pair programming with rotation
Thorough kick-offs
Tech huddles

We’ll get more context-aware, earlier feedback from more people. Over time, pull requests will become trivial – your team will be aligned before the pull request, reducing delay and rework.

Now, we can point out that the feedback achieved through continuous collaboration appears to be more useful than the feedback received in pull requests. If we were comfortable merging solo work with 2 pull request reviewers, shouldn’t we be even more comfortable merging paired work with a single pull request reviewer? What if we rotated pairs – isn’t the context-aware feedback of 3 close collaborators worth more than 2 peers with minimal context looking at a diff?

Stop branching

By now, we’ve already:

Built quality into our practices and codebase so that we feel confident integrating our work
Learned how to integrate more frequently by working in small steps and using feature toggles
Discovered that collaboration is more effective than critique and broken our dependence on pull request reviews for feedback

We’re ready. All that’s left is to be brave and ask our team if branches and pull requests are still adding enough value to justify the delayed integration and context switching they cause.

Collaborating Continuously

July 28, 2021/Eli Mydlarz/Leave a comment

Some engineers have found close collaboration (by pairing, for example) quite challenging and feel that they are more productive when working on a branch by themselves. In a team setting, this impression of higher productivity is mostly illusory.

Ultimately, the people on your team need to understand, approve of, maintain and extend your work. If you don’t come to some understanding together during development, you will have to work it out at the end in a slower process with less shared context and more rework.

Most challenges experienced in close collaboration are temporary. With practice, pairing skills will improve, technical knowledge will be shared and ways of thinking and working will be exchanged. Your team will converge on effective patterns and practices, greatly simplifying collaboration.

So take that tricky pairing session in stride. Your team will get better at it and it’ll pay off.

Deadlines

March 13, 2021March 13, 2021/Eli Mydlarz/Leave a comment

Recently, one of the engineers on my team asked for a deadline. I refused to set one, then the CTO jumped in and also refused. It made me very happy and I bet it doesn’t happen at many companies, so I want to share why we did that in the hope that it convinces you to stop setting deadlines too.

My Perfect Team

Let me tell you about my perfect team – I’ve been on it several times.

Developers, a business analyst, a product owner, a QA and a designer sit together around a big table with no walls, dividers or big monitors in between them. We’re just a few people with our laptops, collaborating closely every day and pairing with each other across all roles.

We know how to iterate and have scoped out the earliest usable version of our product and the right level of fidelity.

We’ve written great stories according to INVEST principles and prioritised them very carefully so that we’re working on the most important things first.

We’re united by our shared commitment to a common purpose. We care about what we’re doing and work at our best sustainable speed – challenging ourselves but not getting stressed.

Our team is vertical and autonomous – we have all the skills and authority we need to achieve our goals.

Setting a Deadline

Now that you know our team, I want to ask you a question. What will you gain by giving us a deadline?

Will we go faster? We’re already going at our best sustainable speed. Maybe people could work longer hours, but we only have so much mental energy to use and our work is intellectually demanding. Even if we do manage to get more done for a while, it will increase turnover and that will cost more than its worth.

Will you actually have any more certainty about when our work will be finished? You’ll have a date, but in your experience, have those dates really been reliable?

So we trade some possible short-term speed for increased turnover and we get a magic date that – if we are honest about our experience – we know we should not trust. And what will it cost?

Well, you’ll almost certainly guarantee that the work will not be delivered before the deadline.

For the deadline to be credible, the team will have to conduct regular agile estimation and velocity measurement. That will take a lot of time, delaying delivery.

The external pressure will impact the team’s sense of autonomy, further damaging morale and increasing turnover.

Finally, external knowledge of the deadline will create a false impression of certainty that encourages timeline based programme management. We’ll end up with a fragile, interdependent series of timelines that falls apart as soon as the first deadline is missed.

Move On

I know it’s hard to let go of deadlines, but we have to move on. Accept uncertainty. Stop being obsessed with what we can’t control and focus on what we can.

Replace talking about time with talking about scope and fidelity
Replace velocity measurement and estimation with better story writing and story slicing
Replace timeline setting with prioritisation
Replace external deadline pressure with shared commitment to a common purpose
Replace timeline-based programme management with creativity and concurrency

Pragmatic to a Fault

November 10, 2020April 8, 2022/Eli Mydlarz/1 Comment

In the last few years, you’ve probably heard these phrases a lot.

Can we focus on the problem in front of us?

We need to be practical here

Let’s try to be pragmatic

We’ve all worked with engineers who care more about applying SOLID principles than delivering working software. As an industry we’ve been reacting to that – and rightly so. But I have also seen this language used to shut down conversations that – though they may have seemed insufficiently pragmatic at the time – turned out to be very important. We push aside conceptual problems to focus on practical problems, but today’s practical problems are often symptoms of the conceptual problems we pushed aside yesterday. Pragmatism can become a shortsighted cycle of conceptual neglect.

Conceptual Problems You Should Care About

Some conceptual problems can be kept in our peripheral vision until we understand them better, but others need to be addressed right away. How do we know which is which? To try and help, here are some of conceptual problems that have caused issues for my team. When we see one of these now, we tackle it right away.

Inconsistent Thinking

If your code reflects different ways of thinking about and solving the same problem, clashing mental models will cause confusion sooner rather than later.

In publishing certain messages for downstream systems to consume, we needed a mechanism to identify which billing period each message related to. Different messages were implemented by different people over a period of several months and when I examined the whole subsystem at the end, it was clear that the logic for finding the billing period had been invented anew by each engineer. Depending on the message, it was sometimes being found by the bill, sometimes by a created-at timestamp and sometimes by a combination of the two. There were 3 different strategies implemented across 5 different code paths.

When I expressed concern, I was encouraged to be pragmatic – after all, the integration was working fine. But we have enough users that errors in thinking quickly manifest as mishandled edge-cases, so I wanted to get out in front of this one. I began converging our various code paths and strategies around the most sensible solution.

Just before I completed the job a Product Manager in another team sounded the alarm – their numbers weren’t adding up because certain messages couldn’t be placed in billing periods and it was costing us money. Because we had begun solving the conceptual problem instead of waiting for a practical problem, we were able to resolve the issue on the same day and avoid serious financial harm for the business.

Self-Defeating Compromises

We make trade-offs to get things to market faster, but there’s no point delivering something faster if the core value remains unrealised.

One of our teams developed a service for centralising ownership of certain account information. Downstream systems had been designed to expect this information with a special key, but so far only one key had been used. Developers of the new upstream system decided to save time by only supporting that one key.

But downstream systems weren’t designed to expect a key arbitrarily – being able to differentiate these records by key was essential to providing a good user experience. As soon as the business added a second key, the upstream system was rendered useless. The compromise was self-defeating, in that it undermined the core value of the work.

Misleading Behaviour

Anything that doesn’t behave the way it leads people to think it will behave is an expensive mistake waiting to happen.

A member of our team named the primary key column for a new fees table transaction_id. As the team already had a ledger table called transactions and a convention of storing off-ledger supplementary data in other tables (with transaction_id as the foreign key), the naming decision misled many engineers into thinking that the new fees table was another example of this pattern. Much time was wasted explaining to people why they were not able to find those transaction_id values in the transactions table. When the misunderstanding began to influence people’s design decisions, we stepped in and changed it.

Lost Information

We all know not to throw out important information. But many engineers conflate what happened with their interpretation of what happened, discarding the former and only storing the latter. I wrote a whole post on this topic called Write Now, Think Later.

The Real Cost of the Panama Papers

August 6, 2019August 6, 2019/Eli Mydlarz/1 Comment

On April 16th 1953, U.S. President Dwight D. Eisenhower gave his now famous speech The Chance for Peace. It’s a wide-ranging speech, but the part usually recounted – and closest to my heart – is as follows:

Every gun that is made, every warship launched, every rocket fired signifies, in the final sense, a theft from those who hunger and are not fed, those who are cold and are not clothed. This world in arms is not spending money alone. It is spending the sweat of its laborers, the genius of its scientists, the hopes of its children.

The cost of one modern heavy bomber is this: a modern brick school in more than 30 cities. It is two electric power plants, each serving a town of 60,000 population. It is two fine, fully equipped hospitals. It is some fifty miles of concrete pavement. We pay for a single fighter plane with a half million bushels of wheat. We pay for a single destroyer with new homes that could have housed more than 8,000 people.

Most of us already know what opportunity cost is and we’re probably not surprised to learn that heavy bombers are expensive, but something about Eisenhower’s direct comparison reaches out from the past and shakes us.

It’s been years since we first heard about the Panama Papers. The world’s wealthiest people were caught hiding their money from the rest of us. The numbers were unreal – so large it was hard to place them in our reality. Trillions of dollars passed through Panamanian firm Mossack Fonseca and while there have been some consequences the matter eventually slipped out of our collective consciousness, untethered as it was to our daily reality.

You can probably see where I’m going with this.

Doing the math

Where do we start? We need to know how much money was hidden – so far so good, the Panama Papers have that covered. Then we need to know how much tax would have been collected – that’s super tricky, because the amount of tax that would have been paid on money had it not been hidden depends on too many variables to name. Finally we’ll need to articulate what else that money could have paid for, which varies wildly depending on where you spend it. Most of these complications arise from the global nature of our problem space – Eisenhower had it easy!

Let’s skip ahead a couple of steps using the prior work of Pierre Moscovici, European Commissioner for Economic and Financial Affairs, Taxation and Customs. Moscovici estimated that offshore tax shelters like those revealed in the Panama Papers cost the public around €1,000,000,000,000 annually. That’s one trillion hard-to-really-comprehend Euro every year.

The Global Partnership for Education estimates the cost of “a complete education from pre-primary through upper secondary” in the developing world at €5,200 (currency conversion correct at time of writing) – note that this is the lifetime cost of an entire education, not the annual cost. India’s newest public hospital – state-of-the-art and their second largest – was recently completed at a cost of €161,668,600. Finally – for my fellow Melbournians – the cost of building a national-average-sized dwelling on a median-priced lot in Melbourne is apparently €364,600, the average number of occupants per dwelling is 2.7 and the number of homeless in Australia is around 105,000.

With these numbers in hand, we’re ready to look at the Panama Papers in a way we might actually remember.

To borrow from Eisenhower…

Every dollar of tax evaded signifies, in the final sense, a theft from those who hunger and are not fed, those who are cold and are not clothed. Our wealthiest people are not hoarding money alone. They are hoarding the sweat of our labourers, the genius of our scientists and the hopes of our children.

The annual cost of our wealthiest people evading taxes is this: A complete education for at least 190 million children in the developing world. It is more than 6 thousand large, fully-equipped hospitals in India. We pay for tax evasion with new homes that could have housed all of Australia’s homeless 70 times over each year.

Engineering Managers and Ants: The Search for Consistency

July 23, 2019July 24, 2019/Eli Mydlarz/Leave a comment

Have you ever seen (or been) an Engineering Manager or Software Architect looking for consistency between teams? Have you ever seen a trail of ants, carrying food back to their colony? I’m going somewhere with this – I promise.

Imagine (or remember) a trail of ants. The terrain is complex and – for an ant – dangerous. The food source is very small. The ants have a terrible vantage point and can’t get a good look at anything. They’re not very bright, even by our standards. But the path they’ve found? It’s almost perfect! It has some twists and turns, but it’s relatively direct. There’s a dangerous obstacle nearby, but the trail gives it a wide berth. How did the ants find that food? How did they figure out a path that balances safety and directness from such a terrible vantage point and with such tiny brains?

How do the ants do it?

The basic mechanism behind this is well understood and very efficient – so much so that we frequently use it to solve problems of our own. Let’s start with a simplified model of ant navigation:

Ants follow the pheromone trails left by ants
If there’s no pheromone trail to follow, ants wander off in a for-our-purposes-random direction

Drop a colony of ants in the middle of a flat plane with no pheromone trails to follow and they’ll spread out in all directions, each ant leaving a new pheromone trail. When an ant finds food it resets and follows its own pheromone trail back to the colony. This trail is now twice as strong as any other, because it has been traversed twice. Other ants encountering this trail will change course and begin to follow it, strengthening it further until eventually all the ants are using it to ferry food back to the colony.

Now we know how ants find food. It’s an amazing trick – they’re small, stupid (by our standards) and can’t see what’s going on, but with a couple of simple behaviours they can do route-finding in complex environments. It’s a triumph of distributed intelligence – a whole greater than the sum of its parts.

Imitation is the sincerest form of flattery

So I guess we should do that too, right? Teams try all kinds of ideas until one of them finds something that works, then we all converge on that solution and keep doing it (well, until the food runs out). That’s roughly what I’ve seen in many organisations – perhaps you’ve seen it too:

“The other tech leads and I talked about it and decided we should all use <Library> from now on”
“<Manager> wants all the story walls to be consistent, so we rearranged yours a bit last night”
“All the other teams follow <Process>, so we should really be doing it too”

To some extent this emerges naturally – humans are great at learning by imitation – but often convergence on a single solution is driven from the top down. There are plenty of good reasons a leader might nudge people in that direction and you’ve probably heard some of them:

“It’s easier to move between codebases if they all use <Pattern/Library>”
“We don’t want to be constantly reinventing the wheel”
Homogenous teams and codebases simplify resource allocation and reporting (you don’t hear this one spoken plainly very often, but it’s a biggie)

Local optima

So far, so good. But we haven’t explained the ants’ real secret yet! If ants really used our simplified model of navigation, they’d converge on the first solution they found. You’d get a trail that leads all over the place before it eventually happens to collide with something useful. Ants following it would waste unnecessary energy and be exposed to needless risk using a wildly inefficient solution they arrived at almost by accident. They’d be trapped in a local optimum.

As we said earlier, ants generally do better than that. Ant trails are relatively direct, even over challenging terrain. They’re small, stupid and have a terrible vantage point – so how do they overcome these weaknesses and escape local optima when all they know how to do is follow pheromone trails? Amazingly, they do it by being even worse than you thought! As it turns out, ants can’t even follow pheromone trails properly. They regularly get lost and wander off the trail and in so doing, they sometimes happen upon a shorter or safer route than the one they were trying to follow. Ants that move back and forth along this new route will – quite incidentally – end up increasing the strength of the pheromone trail quicker such that it eventually becomes the dominant trail. After enough of this, the ants will end up with something close to the best trail possible – a global optimum (for a suitably narrow definition of the problem). Their inconsistency is a crucial part of their success.

Intellectually we know the risks of converging too quickly on a solution. When we solve problems using particle swarm optimisation or evolutionary algorithms we are careful to keep introducing random variations and thereby avoid getting trapped in local optima. But when you get a few tech leads, architects or engineering managers together we seem to forget all about it. We forget that inconsistency is almost synonymous with innovation – that it is the most effective means by which new ideas can be tried, evaluated and eventually adopted.

Finding balance

There are good reasons to want some amount of consistency, but we regularly take it so far that I suspect there’s more going on. Maybe we want predictability in a stochastic world, control over empowered teams or some other hopeless paradox. Perhaps in our fear of uncertainty and our haste to converge on the safe and the known, we trap ourselves in local optima. Not only does this limit our potential, it is deeply disempowering to the people doing the work. After all, even ants are allowed to wander off the trail!

While a little consistency can be very useful, total consistency is like rigor mortis. No new tools, techniques or ideas. Disempowered developers stuck in the past, feeling like pawns and taking orders from people who think they found all the answers years ago in a rapidly changing world. Developers working under such a regime will – if they have any interest in their field or their sanity – quit and find something more interesting to do.

To find balance, we need only stop meddling. Humans are big imitation learners, we often go along with bad ideas just to fit in and we’ve all heard that good developers are lazy. If despite all of this your teams are still trying out something new, then they probably have a strong intuition that they are in local optima – that a better solution exists somewhere out there. We should respect that intuition and avoid upsetting the balance with top-down pressure to converge prematurely.