As a /senior/ developer I really dislike blanket statements. I've seen the same amount of failures caused by
> “Do we really need that?” > “What happens if we don’t do this?” > “Can we make do for now? Maybe come back to this later when it becomes more important?”
as with experimenters. Every system is different, every product is different. If I were building firmware for a CT scanner, my approach towards trying out new things would be different than a CRUD SaaS with 100 clients in a field that could benefit from a fresh perspective.
There are definitely ways to have eager/very open seniors drive systems into hard to get out corners. But then there are people that claim PHP5 is all you need.
I came to say somethign simular actually.
> Ah, baby, this is my senior developer. The avoider, the reducer, the recycler. They want to avoid development as much as they can.
There are times when this is good, there are times when actively trying introduce an improvement is the best way forward. A good senior is able to recognise when those times are.
I think this is more a matter of perspective, rather than original meaning.
I read the above as "avoid development that increases complexity needlessly" — and often, there is a desire to overcomplicate something that can be much simpler because the understanding is lacking.
"As much as they can" does not mean trying not to do any work, but trying to simplify the work where it achieves desired outcomes, and just about! This frequently means doing the improvement today.
> There are times when this is good, there are times when actively trying introduce an improvement is the best way forward. A good senior is able to recognise when those times are.
This is what I was thinking - I'd say the biggest step up a developer can make is to recognize that sometimes you need a bit of one approach, sometimes a bit of another one.
Sometimes minimalism is the way, and you need to wonder if the pain, workload or lacking capabilities and features are problematic. Or, sometimes adding the smallest possible thing is a good way, as long as we don't paint ourself into a corner and enable learning and accumulating information of what we actually need.
Sometimes buying a thing is a good way, if you can find a good vendor and a tool fitting your use case and especially if the effort of doing it on your own is high. This commonly occurs in security, because keeping up to date with the ongoing vulnerability and threat landscape can be a full job on its own.
And sometimes adding something bigger is the way, if the effort of maintaining it are less than the effort and pain incurred by not having it. Or if we can ramp up the effort of the thing incrementally, while reaping benefits along the way. This can be validated often by doing a small thing.
What the AI will do in my opinion is to push the bar more in this direction. Cozily hacking CRUD-Code in a web server together most likely won't be enough in a year or two for the average development job.
That doesn't sound as good in meetings. The person who can cut scope and get everyone to the "we did it" back patting phase makes everyone feel warm and cozy.
Now combing through analytics to determine whether or not what we did was actually good? Less warm and cozy.
This is where good leadership in the dev team is needed.
Is the improvement likely to reduce maintenance overhead (and thus cost)? Or improve performance allowing for fewer services running (and thus reducing cost)? Or reduce bugs that force people out of a workflow (eg in an online shop, thus fixing it increases sales)?
Or if it’s just tech debt then use Jira (etc) to your advantage and talk about the number of tickets you can close of this sprint due to this engineering initiative.
If the development team and product teams goals are largely aligned then the problem with engineering initiatives is just how you explain them to the product team.
For a large enough problem you need a combination of enough skill (to do the job), enough foresight (to know what likely will go wrong and how much error budget you need), and skin in the game (so you dont just cut things that sound good but instead what is truly needed) - if you don't have all three of these you usually are just talking out of your ass.
both of these things are equally important. every change will annoy somebody. every change breaks somebody's workflow.
preventing the unnecessary changes can help you get the political capital in your org to push through the changes that really need to happen.
I am an avoider and also a serial trend-hopper. You can do both!
Exactly.
A sort of survivor bias. A VP ordered to use elastic search, because it worked well at his company before. Turned out it worked well for us. Listen to the VP to make technical decisions. And use elastic search.
Reminds me when the ELK stack was called just ELK (idek what it is now) we had a server we put it on, and after making the additional dashboards my manager wanted, we learned the limits of ES / ELK. It needs a ridiculous amount of memory, because it will shove everything in memory. Same thing when I learned that MongoDB indexing puts every item in memory as well, which is a yikes, why would you not want to index?
I bet there's money to be made for building a drop-in to either of those two that requires less memory, would save companies a bundle, and make other companies a bundle as well.
There's no high performance database that wont take all of your memory (at least for size of data) if you let it.
That's because it's much, MUCH faster to do it that way, though if you can deal with certain type of latency trade offs for throughput something like turbopuffer can do wonders for your costs.
MySQL doesnt eat up all 8GB of my system when I need to query a table with indexed values, MongoDB seems to eat it all up.
You paid one hundred bucks for that eight gb of ram, do you really want it to just sit there unused?
No, but my manager was wondering why our website was slowing to a crawl.
Is the DB on the same host as the web server?
It is more likely they did not leave enough overhead for the host operating system, which is a classic issue.
I don't really remember, to be fair this was nearly 10 years ago now. Upon some googling now, I do see a way to limit just how much Mongo sucks up for data + index. I am curious if it would have been a smoother experience, if this configuration was even available then.
[dead]
If the data is < ram size and if you read that data again and its off disk again its the slowest it can possibly be, there's a reason most databases implement a buffer cache (actually making writes insanely faster as well) but yeah, MySQL is generally not a very good operational database with all the ones I have tinkered with.
Production grade multi tenant databases want to *solely* run on RAM.
> why would you not want to index?
Because if you don't need an index it wastes RAM, as you've learned. Maintaining indices also has a cost. Index only what you need.
In the sense of the blog post: A senior with decent DB experience would have told you. ;)
You mean NoSQL which is slightly different and nuanced, in a shop that was mostly SQL with the exception of me, the one Junior developer using MongoDB and Elastic, mind you, we got a lot of things done and I learned a lot more about Mongo than I would like.
In all fairness this was my first job a few years ago as a developer, I deep dove MongoDB but I was also one of the only devs using it at this place.
My previous experience with MongoDB had been in college and more limited.
Everything "wants to" run solely in RAM, but we don't have infinite RAM, so a "production grade" database should also be able to fetch data from disk unless this is an explicit tradeoff. MariaDB and PostgreSQL do not require all indices to be stored in RAM. Obviously they can be accessed more quickly if they are in RAM but they are designed under the assumption they will often be stored on disk. It sounds like MongoDB is not, and given the reputation of MongoDB, this is as likely to be incompetence as it is to be a willing tradeoff.
Every serious database that is designed to handle moderate to high traffic, will expect you to have RAM to fit all data and indices. Relational DBs do a solid job if that's not the case, but that also sabotages the efficiency you could get from them. It will work for some time. If it's enough for your, that's fine.
I am not experienced with MongoDB, I don't know if previous comment reports were the users fault or MongoDB's. But one thing is clear to me, complaining it uses too much RAM and not knowing the reasons for it, is a user problem. A common mistake is to setup a DB and expect it just magically does works. DBs are complicated beasts, you have to know how to deal with them.
Potentially a mix of both, though MongoDB was still very young when we were using it. Places like Google were championing it, or rather places that can afford to burn a ton of RAM.
You certainly don't need to hold all data in RAM to serve "moderate" traffic. A modern hard drive can seek about 80 times per second, an optimized RAID array even more, and an SSD tens of thousands, and if we're pessimistic, it takes 10 seeks to service a request. To me a light load means up to about a request every second, a moderate load means maybe 20 requests per second and a heavy load means hundreds or thousands of requests per second. Pessimistically each (read) request takes 5-10 random reads to service and almost every system is read-mostly.
I think these are realistic expectations for most apps. Obviously the likes of Netflix and Uber get orders of magnitude more, but 99.9% of apps aren't a Netflix or an Uber, and you don't have to optimize for scaling until your app is on a trajectory to become one, and putting your database on an SSD already let's you handle several thousand concurrent users with ease.
RDBMS are typically pretty good keeping the frequently requested data in RAM. This disguises the latency of disk access and performance will heavily depend on access patterns. If you serve 1TB of data from a DB with 8GB of RAM and that is sufficient for your use cases, I wont stop you. If you expect low, predictable latency (<1ms) even on a 98/2 r/w system, then it it's not worth the headache.
Of course everything depends on use case and constraints. I highlight the extremes here, the initial confusion was why DBs require so much RAM. Traditional DBs are optimized around RAM, that's where they perform best. You can abuse that, but it's not the best they can be in terms of latency, predictability and stability.
For anything Lucene-based (Elasticsearch, Solr) this was a problem where some of the indexed data had to be transformed for another type of query to be efficient, and it put the transformed data into the Java heap then never released it. I think it was indexed data for searching was read straight from disk and was fine, but analysis queries needed the transformed version?
At some point they added the docValues configuration option per-field to do the transformation during indexing and store it to disk instead, so none of it had to be stored in the heap. Instead what you're supposed to do is rely on the OS disk cache, which handles eviction automatically, so you can run with significantly less memory but get performance improvements by adding memory without having to change any configuration further.
Pick the right use case. It is super awkward, horrible UI for things like log analysis. Use Scalyr instead.
Congrats on being the third top-level comment at this hour, and the first one who seems to have read more than just the headline.
Agree. context matter. As a senior developer you need to understand complexity, risk, upsides and and downsides. Understand the business side. If you are a startup or a big company that is already a cash cow makes a difference when changing a core featrue of the product etc... context context context
One of the side effects of the LLM boom is that it made it a lot easier to tell people that context is important
I think this is contrarian, I found author's point clear in context. Obviously sometimes actions are warranted, but the balance today is skewed in making everything more complex than they needed.
This do not mean we don't develop new product and services, it just means when we do so, we find the path of least overall entropy, it also applies to operations and tech debt reduction.
premature optimization is root of all evil
I think you may be missing the message the OP is trying to communicate.
The qualities were highlighted because they can all lead to better stability.
Why can't innovation bring better stability?
Innovation is change, and change is the opposite of stability.
Innovation can reduce pain though, if the current pain is strong enough. A stable stream of failures in production can be the kind of "stability" you want to disrupt.
Being able navigate change can provide stability in the long term though, at least as opposed to being resistant to change.
Yes, all stability in real life is metastability, it needs a constant effort to maintain. A worthy innovation can lower this effort, or lower the risk of a catastrophic failure.
A complete stability is death.
Resistance to change is very different than reluctance of change.
What are we talking about? Philosophically yes. Factually, no. In the context of a system innovation could be switching from one form that renders in 1 second to another that renders in 50ms. Stability isn't part of that equation.
Is this switching risk-free? Consider all these ancient computer devices that run high-stakes equipment for years and decades without change. An RPi could replace an ancient PDP-11, cost a fraction, consume a fraction of energy, be faster, etc. But it also may introduce new and unknown failure modes.
The important thing is to raise the question and have the discussion. By asking the question, you're not precluding the experiment.
I mean blanket statements are bad and you don't want to be the last company running on Java 6, but all the same, it's equally bad to be the guys using the latest javascript build pipeline that came out three months ago and is undocumented.
Was thinking the same thing, but then i re-read the section and noticed this:
> Yes, yes, of course this is simplistic.
It's an example, put to the extreme, to clearly communicate the ideas. As all things, the golden mean applies, as I understand the article argues for:
> the design of the 'Scale' version is influenced by what worked and what doesn’t work in the 'Speed' version of the system.
It's a tricky balance, and there's a nonlinearity in that it really depends on what technical risk you've already taken on. Like.. clever ideas are like children. A handful are fine, lovely even! But if you have more than you can adequately keep track of or properly nurture that's no good. So best to focus attention on the small number of clever ideas that actually matter for your business--the ones that differentiate you from all the other companies doing broadly the same thing as you.
I mean, sure, reduced complexity is great, but... what about performance?
[dead]