I work at a startup and often deal with every part of the stack--programming, deployment, db, cloud environment, networking blah blah. AI agents have just gotten superhuman at computer use. I will never be able to use AWS CLI as well as claude does. I can spend time poking around in AWS, manually reading logs, looking at routing tables, cross checking with data in db, whatever, or I can give claude readonly access to all of that stuff and it will solve the issue literally 10x faster than I could, write a detailed report I can pass on to whoever, etc. Is it sometimes BSing? yes, but it just keeps getting better.
All that said, skill atrophy is real. Thinking about getting back on the interview circuit is p scary because of it. I dunno wtf is gonna happen.
> skill atrophy is real
It is also both a choice and avoidable.
Atrophy happens when you trade understanding for productivity. Productivity benefits your employer and your KPIs, but not your understanding and your opportunities going forward.
Understanding used to be an unavoidable of the exploration process, and writing documentation used to be the thing that never got done. Now, writing documentation is automatic, but reading it still isn't.
I've been managing a team of interns at my last job, and as they were all equipped with coding agents, they'd deliver complete solutions to problems once or several times per day that would normally take days, or weeks if you factor in necessary learning.
As review of their understanding, and for the sake of my own cognitive debt, I would first ask them the question "How does it work?" and they usually couldn't answer. After a handful of tasks, they started preparing answers. This forth-and-back is still more shallow than getting the deep knowledge chiseled through hours of trial and error, debugging and wrong directions. Spending two hours to find a problem to realize you forgot to save or upload a file, or had an x instead of a y. I look back with nostalgia, and I wonder if there weren't always better ways, and we just get to pick them now. I can literally convert any task, solved or not, into a quiz that brings me towards understanding, in rainbow-colored pig latin.
Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
I’m on the camp of using agents as much as you can, but make sure to understand what they did. However, it will still atrophy your skills, no matter what. It’s just different times now, as on the job you’re expected to deliver faster. And obviously that means less time spending on your craft, honing your skills and etc.
Digging a well using manual tools is a skill. Using a machine to dig is also a skill. They are distinct skills. So if people change the way work is done, skills themselves shift.
Using AI appropriately and efficiently and safely is a skill. Similar to using a mechanized excavator. It can do the work 10x faster but it can also do a lot of damage quickly as well.
Those who don't realize this and not adapt to the new skill set required by the industry will lose their jobs. It will cause a loss/atrophy of prior skills and development of new skills but that is inevitable.
Those who have the prior set of skills do have an advantage over those who don't have them. But the new set of required skills can be developed without having the earlier ones too. The most proficient in the new set of skills may or may not be the ones that were most proficient in the old skills.
> Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
Let’s say you have an incident, and you need a resolution. Do you spend 5 minutes tinking, then solving the problem or send out the AI for the hunt and spend 30 minutes grappling with hallucinations and non-sensical output?
The 5 minutes I’ve seen is usually by someone that will send the complete output to someone else to decipher, while saying they have figured it out. Then they’ll turn into a meat proxy between the team and the agent.
In real life, for systems where incidents that require such rapid responses, you usually have fail-safes an killswitches. When you were designing your system, did you put in place alerts and monitoring to mitigate risks?
I’m not sure if you’re running the latest frontier models. Every single engineer (and myself included!) has mentioned numerous times how nothing beats AI in debugging. My other guess would be that you have non-standard set up that Opus 5.5 doesn’t know off the bat how to debug.
I wouldn’t say the same a year ago, but usually we get preliminary reports during incidents before someone can even auth to debug. Almost all the time, nowadays, the report is correct. We obviously verify it.
"Latest frontier models" seems a bit like acting like smarter reasoning can fill in context that doesn't exist in various systems without guessing. It's convenient because it's always the users fault for not working in the platonic ideal observability setup that can feed everything on a plate. And if you're in that situation how much is the agent adding over incremental improvements?
It can't be both "so easy and hands free anyone can use it" and also "well they must be using the wrong model/holding it wrong if it's not working for them". If there weren't tradeoffs of the different models then it wouldn't be exposed to the user.
> Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
It is far more often the case that you will spend more time getting the answer out of the LLM than you would've gotten yourself. They suck ass at troubleshooting systems, very frequently jumping to completely implausible scenarios that no human would think of as the first thing to try, and almost never asking questions to clarify the situation the way a human does.
This has not at all been the case for me. LLMs (even the more-capable local models) are exceedingly good at troubleshooting when given proper context (clues, hints, a decent system prompt/skill that provides even the barest hint of how things work)
What you're describing sounds more like what would happen if you gave a chatbot without access to the repo as little context as possible and asked it to come up with a solution, blind.
> They suck ass at troubleshooting systems
Given the right tools and instructions I find them to be consistently superhuman. It's unreal.
I have an automated system responding to issues occurring on oceanconnect.ca, and every single diagnosis has been correct for months. Of course I still verify, I tune solutions, I review everything. But it's right. It's not a trivial system. It has a fairly large frontend, a large-scale model processing pipeline on the backend, a fairly complex database, and a queue for getting new data into the system. Claude does a stellar job of navigating and understanding problems, even when they're not 'real' or as they initially appear to be; things like quirks in old versions of Safari, or the model pipeline fails due to data integrity issues from the providers.
I've rarely encountered cases where I need to steer a frontier LLM much when troubleshooting. They're exceptional.
Sorry, this might be a thing in your org, and I have no idea hoe I can convince you, but it’s just not true in our environments. Maybe you might have a very non-standard set up, but frontiers do amazing job.
I’m a troubleshooter and I’ve managed human troubleshooters. The LLMs are better - if they are grounded in either IAC or api access to the infra.
I couldn't imagine submitting a PR for code that I didn't understand and couldn't explain. You can even have AI explain the code to you, so it's easier to understand.
I think a lot of this skill atrophy talk is over-blown. It definitely happens, I experienced it some, but as long as you slow down a bit, and clean up and understand your code, your skills won't atrophy. Actually, AI provides a great opportunity to learn at an even faster rate by using AI to help explain complex topics.
I think perhaps my use of the word skill atrophy here is wrong.
Skill atrophy is when you're good at something, and because you don't practice it regularly, it gradually wanes over years, like when my former CEO said "I used to be good at programming" because he made a tiny chess computer and sold millions of copies, and now he... just doesn't code, and hasn't for decades.
Skill atrophy would be if I opened my text editor and I wasn't able to code after two years. Being a polyglot, there's always some language that feels rusty. And now they all do. But it hasn't been long enough to feel the atrophy.
What I'm describing is laziness to understand a problem domain.
> I couldn't imagine submitting a PR for code that I didn't understand and couldn't explain.
I could. I do it constantly for projects where that premise is acceptable. And I've worked on projects where many people merge to main once an established set of steps has completed. "How does it work?" and "Show me the thing working." are good gates. Understanding (or even reading) the code is not a mandated prerequisite.
As an Open Source contributor? No way. Exactly because I'm contributing to projects where people don't share basic premises. Some people code by hand, others use coding agents. If I were compiling stuff and everyone else was hand-writing their assembly, I'd read and tidy up my generated assembly, too.
> AI provides a great opportunity to learn at an even faster rate by using AI to help explain complex topics.
Totally. Better solutions come through those explainer sessions.
[dead]
The way I think about this -- AI makes all of us managers. We need to figure out when we are convinced by what is brought in front of us, when to ask to do more checks/validation, when to fully reject, when to accept. To do this properly we need to have to have enough background knowledge to evaluate the what is AI brings, otherwise otherwise a person in the loop is simply not needed if they only type a prompt and always click yes.
I think people indeed need to constantly practice our low level skills, otherwise they will atrophy.
Managers or rather inventors and solution providers? We can do a lot more than just manage now.
But there's a new skill blossoming there, right? That's what I'm noticing. The atrophying skill is one you don't need anymore. The new one - prompting, routing the output, spotting the hallucinations, managing singal/noise etc. - is now where you can add tremendous value.
To say nothing of being bold enough to run systems like that in the first place. I know some of engineers that are still folding their arms and refusing to engage with AI!
>spotting the hallucinations
SRE here, spotting the hallucinations requires you to have the knowledge to know when it's wrong. Best way to gain knowledge on systems is actually working on them. If AI do all the work, then you don't get the knowledge.
Exactly. How hard is it to understand for people?
If you don't ride your bicycle for years and years because you can just write "drive yourself"
Then your skills are atrophying. Typing isn't a skill
I don't know that that analogy fits for this comment. The argument is that the old skill is genuinely no longer needed in favour of the new skill, but the new skill requires experience from the old skill. I don't think being a good driver needs you to have already been a good cyclist.
> I don't think being a good driver needs you to have already been a good cyclist.
It's an established fact by now that if you know coding then you can get better output from Sonnet than vibe coders can from Opus
I think it basically stems from the idea of “conceptual learning”. Which has been a pedagogical ideology for quite some time now. And is also convenient if you’re a bit addicted to your phone, a bit lazy due to AI, and want to still feel good about “learning” things.
We’ve known that one important component for learning (not the only thing that matters, but an important one) is the memorization of facts.
There seems to be a big urge to pretend that’s not true. E.g. you can learn arithmetic without memorizing your multiplication tables, literature without engaging with the canon, and now apparently coding without even having coded.
The idea is if we start with a top-down conceptual approach we will eventually vibe our way into knowledge acquisition. Which does sort of work.
However, sorry to be a curmudgeon, but the people who just “do the work” and actually spend time on task to learn a bunch of stuff … are actually more knowledgeable on average.
> you can learn arithmetic without memorizing your multiplication tables
How? If you don't learn multiplication tables you're multiplying on the go. And that's inefficient
Also, this isn't the right analogy. I get where you're coming from.
We move on to the higher form. Like we don't need to code in binary, or assembly language We got C, Python. We then got compilers.
But LLMs are just english. There is no skill here in writing the prompt. "Give me a crud app that does xyz" is hardly a skill
Tomorrow if AI data centers fail then what do we do of the typing skills?
When we shifted from Assembly to Compiled languages or to Python. We still had to learn something else
Here, using AI, its just English.
How do technical managers solve this problem? I've never been a real manager.
Good ones force themselves to do technical work hands on regularly, but will generally reach a level where they can't actually do that anymore unless they are really good at building management systems and delegating. For me it has happened in cycles over the last 25 years; after 4-6 years in management I refresh skills and go back to an IC role because I can't stand it anymore. I've done that cycle three times but I won't do it quite the same way again because its probably impossible to do without a pay cut now, and because IC don't write code anymore anyway.
This is both like forcing all IC to develop management skills but its also like the shift to cloud infrastructure or 4GL languages - we are never going back to the old ways and will be building everything at a different level of abstraction with a different span of control.
The good ones still make time to knock out some lifelong backlog resident tickets to sharpen their axe on.
The bad ones get promoted for spending the time instead playing politics.
I guess humans don't hallucinate that much
Humans hallucinate much more, but they're not that good at making the hallucinations convincing.
No they don't. Any human who hallucinated as much as an LLM does would be committed.
It's interesting how machines now mirror our relationship to sanity.
Just under the surface of human consciousness, we are constantly hallucinating and predicting what we may occur next. We are good at discarding what doesn't align with our working model of the world before reaching awareness and taking action. It's our attempt to conceptualize reality. Considering how un-grounded from reality humans are becoming, the fact we have to balance our nervous systems, the mistakes we do make day-to-day, and how minor physiological changes (from poison to injury to drugs to random chance) can disrupt the process, it doesn't take much for humans to make a major mistake or enter a psychotic state.
It wouldn't surprise me if a systematic approach to solve these problems with LLMs could be applied to human/hybrid teams.
I don't think that LLMs mirror human cognition or "sanity" as much as people want to believe. We are not constantly hallucinating in any way that applies to the term as commonly understood relative to LLMs or to humans. Mistakes, minor psychological changes and emotions aren't equivalent to LLM hallucinations or human psychosis. The relationship is superficial due to both humans and LLMs using natural language. But humans understand that language and its context, whereas LLMs do not.
The narrative exists to justify deskilling and replacing human expertise with LLMs on the premise that LLMs are at worse human equivalent and at best human superior but the hyperbole employed to make the false equivalence sometimes borders on the absurd and occult.
>It wouldn't surprise me if a systematic approach to solve these problems with LLMs could be applied to human/hybrid teams.
What you're describing aren't "problems" that need to be "solved" in human beings. They are problems when they persist in software, but we already had software that didn't have these problems.
This is a state of being that we chose because we decided profit superseded all other concerns.
> I don't think that LLMs mirror human cognition or "sanity" as much as people want to believe.
I don't "want" to believe it given the implications, nor support a profit narrative. That incorrectly assigns a frame of thought that has nothing to do with me and actually opposite to my beliefs.
We do know that a single chemical compound can fundamentally alter our perception of reality. Drugs don't contain hallucinations, they reveal our subconscious brain processes by affecting our awareness and disrupting whatever ends up in our cognition. We are constantly trying to predict what will happen next and compare outcomes to expectations, where errors and confabulations occur constantly. False memories and unreliable eyewitness accounts are a matter of fact. Optical illusions are persistent and repeatable. We're often confidently wrong when recalling memory, deducing from logic, transcribing etc., whether it's for milliseconds or decades. We also dream each night which is technically not "hallucination", but clearly a necessary reality-adjacent process.
If you break the problem down and question our ingrained societal human exceptionalism, it's clear humans actually have no real unshakable "understanding" of anything, we simply have more sensory information and reality testing in real-time to maintain a state of reality alignment. This can be affected at any time by physiological changes from illness to brain injury. Perhaps we currently maintain better context overall (reducing the volatility and frequency of errors), but no animal would exist for long and succeed if it didn't naturally do this one task exceptionally well.
Given the pace of technology and lack of evidence to prove digital machines can't outperform us at this task too, long-term it seems like a matter of time until LLM + ? = human ability superseded. At which point we may have an entirely new tool for neuroscience research, before we even have an agreed definition for consciousness.
This was all evident before LLMs reached the stage they're at now, and progress will continue whether or not companies can capture profit in a hype-bubble. The argument otherwise would be to prove that humans have a deterministic-probabilistic relationship that differs from the technology. I don't have answers but we certainly can't close these open questions when the literature and evidence points more towards similarities.
Not even that Proton CEO claimed that. This is another one of those "no u" knee-jerk reactions littered all over this site, and I wonder when we'll first see stuff like "but humans also sometimes fail to load because of network issues", because there is no real distance between "humans hallucinate even more" and something like that.
> "but humans also sometimes fail to load because of network issues"
They do. It's called "stuck in traffic."
Humans don't literally hallucinate much, but we do lie, or are wrong.
Humans are deterministic and have personality traits. The first couple of times getting bs’d and that person is now a known quality.
Agents fail randomly. One day a stellar dev and next a bs artist.
So this idea that LLM hallucination is not unlike human failings is without merit.
> Humans are deterministic
what
A person has what is known as 'character' and 'personality'. These words indicate discenable patterns of behavior. Determinism in this discussion means that a given person will 'behave in an expected manner'. Further feature of humans in that you can coerce them to behave in highly constrained manners.
Entire institutions and organizations in humanity are based on this feature of human beings.
A heuristic software agent that will respond in a spectrum of possible ways and past performance is no guarantee of future performance. It is no way reassuring that your agent has 'never done x' since if 'x is in the space of possible responses' it can certainly do 'x'.
> Determinism in this discussion means that a given person will 'behave in an expected manner'.
Oh I see. That is not what determinism means. Determinism means their actions are predestined, "determined" by the initial conditions. Computer programs - including AI - are deterministic, barring cosmic ray flips. The same inputs running in the same environment produces the same output.
Humans are or are not, depending on whether free will exists.
Either way, it's hard to draw the boundary around determinism such that it includes humans but excludes AI.
No, determinism in this context clearly means precisely being 'predictable' as claimed that individual humans are predictable given sufficient observation.
It's 1930 again
Every new model and you might as well chuck 90% of that knowledge in the bin.
Last Claude major version they released a blog post basically telling you to chuck away your Claude.md file and start again.
It was only 18 months ago when people were telling you to use ultrathink:
https://news.ycombinator.com/item?id=43739997
And then that went away about a year ago, and now it’s sort of back again but not quite as it was.
I really don’t want to see code generated by Claude without some constraint like rules in a Claude.md file. Even Fable on high is lazy enough to forget a ton of things that a senior engineer would not. For example tracing and handling all unhappy paths, not repeating code, not writing terrible doc strings, etc.
We are using quite a big list of rules and the output is significantly more aligned with what we interpret as “good” code.
[flagged]
I don't know how to spot hallucinations, manage signal/noise, without having a pretty good understanding of the problem space. It's a tough balancing act.
No, everything you listed is low effort and minimal friction. It's exactly the luddite's greatest fear: to be delegated to a button pusher, to ensure that blinking green light continue to blink green. Anyone that can read and write can now add "tremendous value," so what do you think makes the value "tremendous"?
A company comprised entirely of management. I'm sure only great things will come of this.
> The atrophying skill is one you don't need anymore.
of course. who needs to walk when we have cars
This is always a good example
People cannot deny that cars have made our physical bodies atrophy. Not all of us, maybe, but many people are way worse off physically than they would have been without cars
There's an analogy in there but I can't think of it. Maybe I'll ask Claude if it can explain it
Skills that you need come and go. I owned a commodore 64 once and knew how to use it. All gone. I fired up an emulator a few years ago and then had a big "now what moment" where I realized I had no clue what to do with it. I had a vague memory of inserting tapes and issuing commands to load stuff from it. Function keys were important. And there was a manual I once had that explained it all. But no muscle memory. Nothing. I'm sure I could have figured it out eventually. Same with DOS. It's decades ago since I did anything on a DOS command line. All gone.
This doesn't matter. That stuff belongs in a museum. I've used AWS pretty extensively a few years ago. I switched to gcloud and other providers later. At some point in the last year, I started delegating all of the pesky work of interfacing with that stuff to AI. It's amazing. I've always referred to AWS documentation as a maze of misdirection. Just endless amounts of documentation mixed with marketing pages that are completely useless. All links lead back to those. So you are constantly fighting their marketing people to get at the bit you actually want which is a mess of fragmented reference documentation with parameters, endpoints, etc. Staying on top of that and keeping it in your head is a skill that will atrophy that I never cared about.
AI is great for figuring out micro detail like that. It's also great for coming up with a bigger plan that involves stringing together lots of tools, commands, scripts, and what not to execute a high level goal of "I want to run this thing over there". And with a bit of work you can actually do this the right way as well. It's fine. I never really liked devops to begin with. I can do it and I appreciate it done properly and can tell the difference. But I've always hated how it sucked up my time when what I really wanted to do was move our product forward. Instead of which I ended up dealing with a lot of arcane AWS or whatever crap that just needed dealing with. And there's always more of it. Just endless amount of micro detail. AI is great for getting on top of that.
I used to scour through server logs, build logs, test reports etc to figure out why tests were failing in CI. That was exhausting work and could take days of investigation. Now I get the exact root cause in a couple of minutes. Just had to give LLMs a token to call Jenkins API and it knows how to do the rest. What an unbelievable experience. Never ever thought computers would be able to do this in my lifetime.
Had this eye-opening moment a few days ago where I enabled our company-wide MCP server and asked for a problem we noticed. The agent looked through slack messages, logs from our devops portal, open telemetry traces and searched code snippets from other services through our RAG code search tool and created nice tables with timings, explanations and suggestion to fix it.
This is an amazing use of LLMs, contrary to offloading your thinking to it during implementation (which does not mean not using to write code but to actually understand the systems you are creating with it, and all their edge cases, inner workings etc.)
You may never be as fast, but you should absolutely feel at least hypothetically capable of using the CLI at least as well. Or just.. Reading docs and understanding patterns or interfaces was never a superhuman skill and it still isn't. You are a lot more capable than you probably think!
And I am not just trying to platitude you here. Without that basic confidence everything ends up just eventually being harder with the AI, even if maybe the feedback and loop of it all feels faster.
Velocity is sought because we lost traction. If you can effect your designs in few actions, speed has nothing on intention.
This. It seems most developers have never even had the experience of iterating on a system they understand, making predictions, seeing the result, and having a clear sense of what they are trying to accomplish. There are so many tools and systems, wrangling them becomes an end in itself.
[dead]
> skill atrophy is real
Did you tried adding skill.md?
jokes aside, I agree - models are getting crazy good with CLIs and I kinda gave up learning all the syntax on every tool etc. it makes no sense - what I usually try to do after some _serious_ fkup/incident is to write a human-redable summary and some deterministic scripts which help to debug/fix the problem; for the future when I will eventually ran out of tokens, lol
but aside that I think "the skill" is shifting and a lot of stuff becomes irrelevant indeed; more useful seems to know how to setup your agents so they have access to relevant data easily to investigate and secure your infra and so on; e.g. recently I /llms/ spotted major misconfiguration in reverse proxy setup after I gave it access to all terraform/ansible stuff and let it just probe the hosts/network
I am in a similar situation but I have a different take. Skills themselves are shifting. Where the world is going, professional skills will be closer to what you are doing now. In my view, the valuable part will be to become the person who is responsible. How you do your work, in terms of how much AI you use, will likely become less and less material (in the sense that it will not be an issue if you use a lot of AI to achieve your goals).
Obviously, those who use little or no AI will not be able to deliver as much as those who do. But as long as you become the person who is responsible and show a track record off mitigating things quickly and professionally and able to take responsibility, it will accelerate career progression really fast.
There will be exceptions, of course, like in systems/software where lives are dependent on it or laws are super strict on the software working exactly as expected, but those are going to be exceptions that prove the general rule.
> I dunno wtf is gonna happen.
me either but i do know it will happen sooner than i expect
I’m doing interviews as a senior sre and frankly it’s extremely weird.
I’m being asked very open ended scenario-based questions, I’m answering honestly, and it seems to be going well (i’m moving through the interview funnel).
I’m not getting asked deeply technical questions. No weird obscure trivia. No stack-specific things. No linux internals. No weird networking trap questions. Coding part is essentially gone.
My interviewers so far have mostly been other senior SREs.
It looks like a market where a senior sre can recognise another senior sre. Some kind of closed circle.
So weird, so so weird.
It's a vibe interview :)
Tell me more... I'm also in the same boat, and part of me feels more prepared to do high-level open-ended exploration as opposed to code-based testing.
What kind of questions get asked?
Was the main focus of your job genuinely using the aws CLI and pulling logs though? I was pretty good at it and jq but the only advantage I ever got from that was more time to deal with the problem on the ticket which has been 50/50 in my experience unless the ground is prepared with context + dev environment etc.
>> I will never be able to use AWS CLI as well as claude does.
Are you sure that it's superhuman; and not super -you- man?
That's basically it in a nutshell.
All the people who think that LLMs are superhuman weren't very good at their jobs to begin with.
Yes, LLMs will be able to do things faster than a human. They're computer programs, that's the point. But LLMs only produce higher quality output than the humans who didn't learn how to do the tasks well.
Ah yes, it's just everyone else that's stupid; not you. You're still more capable than Claude right?
And just in case you haven't used the latest AI models and are being genuine, no it really is superhuman. Even if you are an AWS CLI expert, nobody in the world is going to be as fast as AI. And of course nobody is an expert in everything. You're always going to find some random domain where AI is read all of the research papers on it and you haven't.
Who said anything about anyone being stupid?
And yes, I'm more capable than Claude. Every human with normal mental development is. It's common these days to underestimate human intelligence, but being able to churn out lots of AWS CLI text is an achievement measured against itself. One-year old humans learn to speak without having ingested an entire civilization's textual output: adjust your metrics accordingly.
AWS CLI‘s API surface is huge, it’s all of AWS. You can abstract over that with your own tools. Agents are great at these little greenfield scripts, both building and using them.