Michael's Workbench

I had a conversation about best practices recently and decided to write down some thoughts on the topic.

TL;DR

Best practices are like training wheels – they are helpful, but they won’t help you win Tour de France. Current norms were created by breaking away from previous norms. Stick to best practices where you don’t care to stand out, be the rebel to create your spotlight moments. And always keep an open mind.

Why best practices?

Best practices don’t make you the best; they only help ensure you’re no worse than average. Face it, you’re doing what everyone else does – something “tried and true”. This won’t make you a rockstar, but you won’t be the guy who walked out of his house with no trousers on, either. This is not innovation, it’s not moving you forward (unless you were worse than average before), it just keeps you in the pack with everyone else. For most things, this isn’t as bad as it sounds. It’s hard to be innovative in absolutely everything, so for the areas of the business, technical stack, or anything else where you don’t want to differentiate yourself, following best practices is a good idea, it keeps you from falling behind.

Why not best practices?

How did the current best practices come to exist? Simple – someone looked at how things were done and said, “Nah, time for something new”. Maybe the old best practice stopped making sense. Or maybe this person had a revolutionary new idea, or a significant incremental improvement. Or maybe technology has placed something previously impossible within reach. In the data space, we have plenty of examples. In early 2000s, ETL was king and the best practice of running transformations once a week was just getting replaced with doing it once a day (the once-a-week itself was an improvement over the previous best practice of once a month – from two decades earlier). Well, my team had customers who wanted their data within minutes, not hours. We had a choice: build a separate reporting stack for near-real-time, or find a way to turbocharge ETL. We did the latter, coming up with a way to split ETL process into a series of continuously running loosely coupled processes. Today there’s a term for that: trickle-feed, and it became a well established best practice for a number of situations. (I do not claim that we invented trickle-feed, that would be ridiculous. The idea was in the air, and I since talked to a number of people who did very similar things at roughly the same time.) The point is that best practices evolve, they are not absolutes.

How do you decide what to evolve?

This comes from understanding why the current best practice is the way it is. If you’re following a practice blindly, doing things because “we’ve always done it this way”, the difference between what you do and what the followers of a cargo cult do is negligible – you might as well be building a radar out of straw. That may sound harsh, and it’s intended that way – blind following of best practices is BAD. First, it can get you into situations where you’re applying what you consider a best practice to the wrong situation, and causing harm. Second, most innovations build on top of previous practices and results. Without understanding those, if you do innovate, instead of creating something better you run the risk of repeating others’ past mistakes. There is a reason why those who work in cryptography roll their eyes when someone with no experience in that field claims to have “invented” a new encryption method.

Once you do understand the rationale behind existing practices, it becomes easier to see which of the reasons no longer make sense, don’t apply to your situation, don’t reflect the current state of technology, etc.

Evolutionary risk

Evolving or replacing a best practice naturally involves risk. Following the best practice safely keeps you in the middle. Breaking away from a best practice can move you forward and give you an advantage – if you are right, and the change you make is a good one. It can just as easily set you back and make you below average – if you are wrong and the change does not apply to your situation. It’s also important to distinguish between being wrong and the appearance of being wrong. The appearance of being wrong comes from the “naysayers” who resist getting pushed out of their comfort zone. The previous best practice is something they know, and they resist the change – it’s risky, they don’t want to learn new way of doing things, “we’ve always done it like this”, “we tried this before and it didn’t work”, etc. Distinguishing between “we’re wrong” and the appearance of being wrong can be quite difficult, especially when emotions run high. The risk of shutting down an innovation that can take the company forward can be just as high as the risk of going down the wrong path with an innovation. Cut through the drama with hard facts, whether it’s the “5 Why’s” approach, or your own secret ninja way of getting a reality check between rebels and the “naysayers”.

The mix

I mentioned this earlier: best practices have their place – in areas you don’t care too much about. You can’t innovate in absolutely everything – that leads to having more “change initiatives” than people who can work on them, “flavor of the day” changes, and the organization in constant state of flux. The people on such teams are confused about how things actually work (because they change all the time). This in turn leads to high turnover. Which in turn leads to loss of important “tribal memory” of why certain things were started, what state they are really in, etc. If the team (or company) recovers from this, it’s usually by going back to established best practices, eliminating all the change initiatives – with the obvious downside of “throwing the baby out with the bathwater” – killing the initiatives that truly were good, and that would have made a lot of positive impact.

Steer clear of the chaos by focusing the innovation where it matters: the areas that you truly want to differentiate yourselves in. Use established best practices to keep the other areas working well enough. And keep an open mind – the areas that you want to differentiate yourselves in may change over time – remember how AWS grew out of what was just infrastructure for an online bookstore?

What do you think? What was your good and bad experience with breaking away from best practices, and how did you deal with “we’re wrong” vs “they only think we’re wrong”?