A bit of an aside, but after someone introduced me to the notion of Reversible Decisions, it quickly became apparent to me that the solution to the bikeshed problem is to throw money at it before the roosters can start preening about which color the shed should be.
Decisions that are reversible should just go with the instinctive answer of whoever volunteers to work on it.
I've been in many meeting rooms where, because of the number and caliber of people in the room, we've blown $5000 worth of combined salary arguing about basically nothing. I've been in a few where that number was well over $10k.
If you're going to assign a relatively medium talent engineer to solve a problem, it's cheaper to let them solve it twice, maybe even three times, than it is to try to figure out what the right solution is before touching a keyboard. It helps them grow to give them that autonomy, and more importantly training your team out of reflexively reaching for optimization for every single feature saves gobs of money over time.
The interface for a piece of code matters to everyone. The internal implementation details mostly matter to the bus number on that code. If they're happy with it, that matters a lot. That can be overridden by the consequences of that design, but I've seen a couple cases where the bus number for a module wanted a solution with fewer consequences but the group wisdom wanted something flashier but also more brittle.
This reminds me of my experience as a senior contractor joining teams that seem to enjoy endless discussion. For example, when implementing a new feature using an open-source library or framework they will want to make a prototype of all possible libraries, make a report about each of them, have some big discussions about which to choose, and only _then_ start the real implementation. My advice is always to just pick the one your gut tells you is the right one -- if the prototype works, then that _is_ the implementation and we can stop considering alternatives unless we have a good reason to reconsider them.
If a decision is hard to make, its often because the differences are small, and therefore it doesn't matter much and you might as well decide by dice-roll.
> If a decision is hard to make, its often because the differences are small, and therefore it doesn't matter much and you might as well decide by dice-roll.
My life became soo much easier when I had that insight. I used to spend so much mental energy on decisions that most often were inconsequential.
I'd like to know where you're getting these decisions. All of my hard decisions are due to not knowing what technical obstacles we'll run into later.
That is often a different type of difficulties. I was thinking about decision like, should I buy car A or car B. I've done my research. Economy, safety, et c. are very similar. But it is a lot of money, and I don't know which is the "best", and of course I want the "best". I could spend weeks on a decision like this. Now I just acknowledge that they seem to be equal. Car B comes in a nice shade of red that I really like, so I buy car B.
Regarding your question, there's always going to be known and unknown unknowns down the line. My only advice in that case is to get to the unknowns as fast as possible. And from the start take into account that you most likely will have to rebuild a big part when you learn more. That is one of the reasons modularity and decoupling is so important. You want to be able to easily scrap and replace one part without it affecting everything. And keep it simple. Never try to build something that should be capable to handle all imaginable scenarios in the future. You'll be bitten by one of the unimaginable ones, and your complex "can handle everything" architecture will be too rigid to change.
I don't know what the black art of tool selection is, but I'm at least a couple std deviations above average on picking libraries that have legs. Nobody gets it right every time, but some people seem to get it wrong every time.
Be sure the work is given to someone who puts some intention into tool selection and you'll probably be okay though. We don't need three different tools doing the same kind of tasks. But sometimes the old old stuff gets moved to the new tool because the old tool couldn't handle that scenario.
> but I've seen a couple cases where the bus number for a module wanted a solution with fewer consequences but the group wisdom wanted something flashier but also more brittle.
Only a couple? This is some peoples entire job, we can refer to them as "enterprise architecture".
Lived experience is a decade ago, and you once saw Kafka solve someone's queuing problem so now everyone's api interface has to run over Kafka. Even synchronous ones.
Let me rephrase that. I've seen a few cases where it became a major issue, rather than just being a fact.
Often people are smart enough to not ask questions they don't want the answer to. If the three people trusted to handle a problem all are in agreement as to how, then the peanut gallery may just be confounding issues in order to pad resumes or to feel important and involved.
> If you're going to assign a relatively medium talent engineer to solve a problem, it's cheaper to let them solve it twice, maybe even three times, than it is to try to figure out what the right solution is before touching a keyboard.
What’s even worse is when a lot of talented people spend a lot of time planning a solution on paper which turns out not to be workable in practice, and then they throw good money after bad trying to get the unworkable solution to work, rather than admitting they were wrong.
This is what usually happens if you let incompetent people make design decision: they stay with the first and worst version forever.
Right now I'm in the process of leaving a company where this happened over and over again. The company's product and design aspects were always neglected in this specific way: whenever a problem was discovered its solution was assigned to whoever would write the code to solve it. That's it. The QA would be notified a week or two before the release and after the solution was "finished". The QA would then not be allowed to even comment on the overall design, only the superficial bug reports were allowed to go through.
Needless to say the product's code-base is a dumpster. The morale is low because it's a daunting task to deal with this dumpster on a daily basis. Now, even if you wanted to, you wouldn't have a budget to fix the previous design mistakes because of the layers of more mistakes that were added on top of them due to the low morale and lack of budget / lack of procedure for making better design decisions. The only thing that keeps the product afloat is its uniqueness in a rather niche field and the backing of a large company that acquired it, but doesn't really depend on it and has no time for a thorough audit.
> This is what usually happens if you let incompetent people make design decision: they stay with the first and worst version forever.
This is what usually happens when you confuse the decisions that mattered from the ones that didn't. It's like back before 'just use a linter' was the solution to code formatting. You'd start a project, you'd have the 'code style meeting' and everyone would bash themselves into the rocks of whitespace and bracket placement, and never get to things like code organization, structure, and naming of things.
So then you end up with code that uses the same noun in three places to mean 3 different things and uses it as a verb in a fourth.
You only get so many opportunities to really change the direction the boat is steering and you can wear out your welcome trying to exceed it. If you get some things right they'll let you do a few more.
Well, I had to be more specific instead of trying to be dramatic...
The thing is, nothing in the suggested procedure addresses the problem of how you are supposed to distinguish the decisions that matter from those that don't. Whoever came up with this idea must've thought that it's trivial to the point it doesn't need solving, but it's the opposite, unfortunately.
Also, FYO, linters aren't responsible for code formatting. It's the formatter's role. Linters advise on style. I.e. the problem of bracket placement is not solved by using linters, an example of problem that is solved by linters is the maximum number of properties a class is allowed to have.
But the meetings about formatting or using / disabling linter rules still happen. Having tools to do that didn't make the problem go away. What did happen, however, is something more like natural selection. The increased volume of newcomers which the field saw up until maybe five or so years ago were mostly steered in a particular "winning" direction. So, today, naturally, you have most people agree on a set of rules to follow because the majority came from the same background. You need a bunch of old-cadgers to work together on a project for there to be a feud over styling rules.
I would have assumed code reviews would have sussed out those issues earlier in the cycle?
To comment on my concrete (and quite miserable) situation, here's how it went (and does, even worse with the advent of AI, to this day):
* A report comes from the field (to the support team) that something is broken (or highly desired by the customer).
* The support team assigns the ticket to the lead engineer.
* The lead engineer writes the code and after a quick battle with CI pushes it all the way to the customer in need.
* Since it's the lead engineer, he has the authority to merge PRs w/o consulting anyone, which is what he does. You just pull the changes and marvel at them, or at the sunrise, whichever you like best. The "urgency" is used as a justification to skip the due process.
Now, when AI came into play, two horrible things were added into the mix:
* AI audit and code review. The audit is a 50/50 chance between finding a real problem vs misunderstanding of the purpose of the code or the context in which it is executed. The AI will also come up with a solution that is either completely wrong, unnecessary or touching too many things for a human to track. Code review acts in a similar way, except, technically it fills the role of a human reviewing the code, so it gives a sense of false security to anyone making changes.
* AI generating code that is hard for humans to follow. First and most obvious problem is the volume. In minutes AI generates changes that will take months of effort to review. But this is only the beginning of the problem. When reading code written by humans, another human builds a model of the author's approach, ability, intention and permissions. When it comes to AI, these... let's call them "dimensions" are all over the place. The generated code you read may seem very plausible for a while until you stumble into a crucial functionality that completely invalidates everything you've learned about the change so far. The change may make unnecessary segues into the territory it was never meant to touch.
So, even when there are code reviews, they now miss a lot more than they used to when only humans were writing code.
Also, and this is a pathologically bad, but a very common practice: the reviewer's name is missing from the record. You run git-blame and only see the author, not the reviewer. When things go down, the author takes the heat and the reviewer is nowhere to be found. Subsequently, reviewers don't feel like they need to care as much about the outcomes.
Oh, and one more thing. To be effective at reviewing anything you'd want to try things... as in to test them. Perhaps come up with some idea about what things must happen and what things absolutely must not. But where do you find those? Usually... in a design document, s.a. a PRD, but in the case where that design (allegedly) was solely in the head of the person writing the code, how do you even know if the behavior you conjecture the code has (because there are no tests) is the desired one?
+1! I've fallen in love with many of Amazon's in-group concepts, and maybe I'm just drinking the koolaide, but they have the concept of a "two-way door", which is exactly this -- a decision that can be made, unmade, remade, etc relatively cheaply. If you can identify that a choice isn't very dangerous, you can focus on the things that really are instead.
Amazon unfortunately downplays the value of ethics over numbers, and we see how that has turned out. The numbers say that counterfeit goods are just fine.
To be clear, I'm not endorsing Amazon writ large, I'm just saying I like the framing (ha!) as "two-way doors". This is maybe equivalent to your concept of reversible decisions, just by another name.
While we are griping about Amazon, my main complaints are about the fulfillment centers/working conditions, the anti-union activity, building their own versions of products that sell well on their platform, and the penalizing of companies that price higher on-amazon than off-amazon.
Amazon's leadership principles are fantastic. They are applicable to anyone doing any job at any level.
... so were "HP Way", "Siebel Principles", "Hyperion Essbase ethics" and many others, collecting dust on library shelves while MBA students getting instant high from Jack Welch's "Winning".
This often massively discounts the cost of reversing decisions. People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them.
I worked at a large, publicly-traded multinational where decades prior and they were still just a 4 man startup they decided the database server and all timestamps should be in the local timezone.
They are still using EST today even when they have global sharding of their customers/databases between US, EU, LATAM, SEA...
--- you're also assuming that the product roadmap will afford your engineers any time to build it the second, third, fourth, etc. time.
There seems to be some general pattern here that you can find pretty often in "dev war stories" contexts:
(1) We're a small startup/new product team/etc, let's just build the MVP and keep everything simple!
(2) Now we're not small anymore and suddenly have all kinds of nonfunctional requirements we never imagined before! But our simple architecture from before is making everything a pain now!
The natural instinct is then to compromise on the "simpleness" of the first prototype and already try to anticipate all the scaling and nonfunctional requirements that might come later - but that rarely seems to work, as you can't really how (and if) the project will grow.
Seems to me, the real question here is why those teams are still using the "MVP" code even after being well inside the "scale up" phase. Shouldn't this be the point where you gradually migrate to a codebase that is more manageable at scale?
> The natural instinct is then to compromise on the "simpleness" of the first prototype and already try to anticipate all the scaling and nonfunctional requirements that might come later
This is a form of Second System Effect Brooks wrote about in 1975.
https://en.wikipedia.org/wiki/Second-system_effect?wprov=sft...
In addition to Second System Syndrome that has already been covered, I think there are several other things going on here that lead to the tug of war.
In another reply I already mentioned a flavor of Sunk Cost, which is resistance to any change to the code that 'made us successful' because they consider they have already won and taking it away now is some sort of revisionist history effort instead of just Progress. Hofstadter's Law, which tells you we still have plenty of time, despite three initiatives already having finished long past the point of maximum tolerable pain already.
But more damningly, I have worked with a lot of people who want to skip from Make it Work straight to Make it Fast without Making it Right in the middle. Those are the people who, among other things, reach for caching too early, and then declare the war for cost reductions over before they've even started, because there are no perf analysis tools that can see past and around the caching logic to find the issues that caching either didn't fix or made much much worse.
I'm not advocating for making hot decisions at every point along the way. I'm advocating for people having enough experience and foresight to understand which 20% of the code flow is Architecture from which is just feature factory work and making sure that 20% gets 80% of the design thinking and timely preventative maintenance.
On a couple projects I've just done all that myself. But it is a quick road to being a bottleneck and then to burnout. You have to aggressively recruit people to be bus numbers on all of those things. This is part of why I do so much mentoring.
> I'm advocating for people having enough experience and foresight to understand which 20% of the code flow is Architecture from which is just feature factory work and making sure that 20% gets 80% of the design thinking and timely preventative maintenance.
This is true until you're at a company that is a feature factory and nearly exclusively hires out of universities and struggles to work industry hires into their culture. And these tend to make up a glut of the companies a rung or two below the FAANGs/MANGO.
>> People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them
Because for maintenance there is a different team and budget. Same goes about the meetings. Recent example: it's okay to blow $1000 in man-hours to discuss and reject an increase of the compute capacity of my virtual workstation. I bet the cost associated with request was ~$25 of increase to the monthly bill. But it is from the other budget, so it was refused.
I've been doing this a long time and I've rarely seen a separate team and budget for maintenance.
Except for "all of the original team left and we're just here to keep the lights on."
I think it's important to distinguish between situations where the business intentionally made this decision from ones where it was forced upon them. Because the delusion of thinking we can still go on after we scared off all of the people who made this thing leads to a very different dynamic.
> This often massively discounts the cost of reversing decisions. People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them.
This is often because no thought was put into making decisions reversible. Version control, CI/CD, virtualization for example can make it easier to reverse decisions and make it faster and easier to replace bad designs.
I sort of agree. But I kind of think it misses the point.
Bike shedding means no decision is made. That is the worst possible outcome as it provides absolutely zero value. Even a wrong decision provides at least some value even if it fails horribly. Maybe we paint the shed black and it burns down due to excessive heat, but it’s a bike shed so who actually cares if we need to rebuild the entire thing and paint it white this time.
And when it comes to bike shedding, getting it “wrong” is by definition already low stakes. It just doesn’t matter a whole lot.
Even the timezone example isn’t really that material IMO. Sure, it’s an extra annoyance to normalize timestamps to locale in every app/debugging query. But in the end it’s just not that big of a deal.
Worse would have been that early team arguing about it for 6mo without implementing anything. Then no one would be employed currently to curse those early developers to begin with.
Often times the only wrong decision is not making a decision. Very rarely are there decisions to be made that are not overtly obvious which are actually material. These exceedingly rare situations are where the winners from losers get sorted out - but that sort of decision in my experience so far happen a handful of times during an entire career.
As I moved up the ladder in my life I’ve often found my job is to basically flip a coin when a decision choice is presented to me. Very rarely is the topic worth me spending any material amount of mental energy on.
> Bike shedding means no decision is made. That is the worst possible outcome as it provides absolutely zero value.
And "bike shedding" is now a term that people hyperbolically apply to any discussion of something the labelling-participant doesn't really care about. A lot of times doing nothing is actually the best business outcome. Waiting until you have more, better information is great.
Lots of stakeholders ask for things that they think matter and then realize two weeks later doesn't and a ton of man hours have already been spent trying to build those skateboards as fast as possible.
> Even the timezone example isn’t really that material IMO. Sure, it’s an extra annoyance to normalize timestamps to locale in every app/debugging query. But in the end it’s just not that big of a deal.
Oh god are you underestimating the long term impact of data not being stored in UTC. For one, it will be a perfectly reasonable thing that your engineers forget this fact and build new systems in UTC, because that's the reasonable thing to do. Also as the years go on your company will make acquisitions of other companies and whatever timezones their data is stored in (hopefully UTC but obviously not always). To properly integrate this data you have to determine for every timestamp whether there was Daylight Savings Time happening at the time on every single object. You also have to figure out all kinds of edge cases around leap seconds.
Otherwise you have a ton of data that's out of chronology. Pity you if that's a thing that matters to you (it almost certainly does).
Also maybe you're lucky enough to be in a position to modify timestamps of all of your objects in all your datastores without significant engineering effort and coordination...but probably not. Obviously the reason the company I mentioned hasn't done it is because they're in that position...
But say you don't want to "fix it" and just want to do this "on every query"...now this process above is something you have to do across maybe thousands of services for potentially every request...
No big deal, he says...
Well you shifted the goal posts. Storing everything in EST vs UTC isn’t great, but isn’t some business ending level problem. It annoys engineers and causes inefficiency. Unlikely to even show up on a quarterly report if someone chose to write a mandate to fix it.
Once you get to the point of bolting on random systems that need to talk in whatever time zones is when you bite the bullet and change it. Yes, that is expensive.
The point I’m making is that if you spend a week dicking around with two junior devs while in your startup phase arguing about EST vs UTC you lose by default. Would much rather have my imaginary employee just choose EST in 14 seconds and move on at that stage. And I have dealt with systems that have done exactly this.
When it becomes a problem I can afford to fix it. Talking about it for more than 5 minutes at the bootstrap company stage is the definition of bike shedding to me. If those are discussions you are having in your weekly dev meetings you have already lost.
This will obviously depend on exactly what you are developing. Payment systems? Probably way more important to get this right vs. some social media app.
The ironic part to me is that this comment thread is basically bike shedding itself! Timestamps are an age old nerd snipe.
Bike shedding means no decision of consequence is made. People wanted to participate in something, and they sucked all of the oxygen out of the room for other decisions.
It's that important decisions were either not made or got insufficient scrutiny that's the problem. So aggressively shutting down bike shedding that is actually bike shedding by picking a solution and moving on or tabling the discussion to let a team of 2 solve it offline is what you're after.
Sometimes though, bike shedding can be a delay tactic. We have a decision we are not ready to make that someone is forcing a vote on, and there's a silent conspiracy to derail the conversation to avoid the vote for one more meeting.
And then there's people learning not to be responsible for anything, either as a career choice or due to previous trauma at this business. So they are asking for too much input so any consequences are not their fault alone. As you say, being a coin flipper in such situations becomes its own soft power. I've worked at a couple places where a handful of people clearly followed my advice solely because they knew I'd say it was my fault if things went wrong. They just wanted a Decider and I sounded like one so they did things the way I wanted them done. Their motivation was to avoid having to deal with being yelled at. I did my yelling in private, instead of turning it into an opportunity for public humiliation.
When you first introduce the idea you'll find a bunch of people think decisions are Reversible which are not, as you say. And the flip side of Reversible decisions is the Last Responsible Moment, which runs afoul of Hofstadter's Law, and people wait until halfway past the last responsible moment.
The key to reversing a decision is getting over Sunk Cost, to start thinking of some code as scaffolding. Scaffolding allows you to get on to other work and then remove it after, because it's either not needed or the 'real' solution has been installed. People get defensive when you propose to rip out their code. Hey that code made us $250k at a time when we were about to miss payroll. Yes. It did. Thank you for your service. But now it's costing us $30k a month and that shit needs to go.
Getting people to figure out that if a decision is important, making it later is actually the sane thing to do, is a challenge. Because many people's intuition is that we should put energy into this now while it's fresh and we have abundant energy. We can 'solve' it and not have it dangling over our heads. But we don't know the right answer yet. We don't know the strength of our tools or the expertise of our coworkers.
The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes.
(And everything should be in GMT unless you can literally point me to a several hundred page treatise on why another time zone is the correct one. Yeah I've worked west coast places that got bought by NY or Chicago companies and it's a clusterfuck if you both didn't use GMT)
Don't get me wrong, I'm a fan of the idea. I pretty much follow a lot of what you're saying without giving it names. I'm just used to people giving lip service to certain ideas as an excuse to move with less friction in their org and end up doing long-term organizational damage.
Thank you for the added detail.
> The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes.
Also agree, but teams use sprints and "the roadmap" as a way to say no to fixing bottlenecks they've created for other teams and don't want to take the time and effort to resolve.
In accounting this is the concept of "materiality". If some asks "is this material?", they are asking if it's worth spending significant time looking at it or deciding.
Remember to * 3 when assessing. Choose wrong, and you do the job, undo it, and do it again.
Always fun when they manage to categorize it wrong 3 times in a row.
One of the hardest parts of my job is trying to convince people where the pool of blood on the floor is coming from. We are forever stitching up the wrong wounds and missing others entirely.
Pain sneaks up on people and they often don't realize they're in pain until someone takes it away. This is probably why Refactoring is one of my favorite tech books. It's a set of tools and justifications for chipping away at that pain bit by bit, instead of tolerating more and more while productivity grinds to a halt.
At the end of the day, if only three people really have to care about a thing and they're fine with it, is it their business to run the project the way they prefer/tolerate? It does matter if the opportunity costs and externalities are affecting other projects. If they don't have time to work on other features because of the mistakes they never correct. If other people's flow is constantly interrupted trying to work around their quirky stuff. But if we don't see any of that, or only rarely, then it's more that my internal architect trying to be bossy rather than lift all boats.