Atrophy has really been an issue in my recent hiring cycles for good senior engineers.
80% of senior candidates I interview now aren’t able to do junior level tasks without GenAI helping them.
We’ve had to start doing more coding tests to weed their skill set out as a result, and I try and make my coding tests as indicative of our real work as possible and the work they current do.
But these people are struggling to work with basic data structures without an LLM.
So then I put coding aside, because maybe their skills are directing other folks. But no, they’ve also become dependent on LLMs to ideate.
That 80% is no joke. It’s what I’m hitting actively.
And before anyone says: well then let them use LLMs, no. Firstly, we’re making new technologies and APIs that LLMs really struggle with even with purpose trained models. But furthermore, If I’m doing that, then why am I paying for a senior ? How are they any different than someone more junior or cheaper if they have become so atrophied ?
I was actually thinking about this the other day while vibe coding for a side project.
I am a lead engineer, but I’ve been using AI in much of my code recently. If you were to ask me to code anything manually right now, I could do it, but it would take a bit to acclimate to writing code line by line. By “a while”, I mean maybe a few days.
Which means that if we were to do a coding interview without LLMs, I would probably flop without me doing a bit of work beforehand, or at least struggle. But hire me regardless, and I would get back on track in a few days and be better than most from then on.
Careful not to lose talent just because you are testing for little used but latent capabilities.
In your scenario though, how do you avoid hiring based on blind faith?
How do I know you aren’t just a lead with a very good team to pick up the slack?
How do I separate you from the 20 other people saying they’re also good?
Why would I hire someone who can’t hit the ground running faster than someone else who can?
Furthermore, why would I hire someone who didn’t prepare at all for an interview, even if just mentally?
How do you avoid just hiring based on vibes? Bear in mind every candidate can claim they’re part of impressive projects so the resume is often not your differentiator.
You invest time into actually looking at applications properly and look at their previous work and ask them questions about that. The curious and learning mind will probably have some projects, because they are not only into the job for the money, but because they have a passion for the job. That's not a requirement, but it distinguishes the great ones from the mediocre ones.
That’s also something any number of people can fake easily. There’s a ton of smooth talkers who can hype up a project and talk in abstract about their previous projects but weren’t on the side of delivering it.
Show code and explain, or it didn't happen. Takes a capable interviewer, of course, to distinguish sweet talk from profound knowledge.
I’m confused by your response. You expect the candidates to show code from their current/previous employer ? Or side projects that most people aren’t allowed to do by employers?
No, not previous employer, of course (unless open source). I expect candidates to show code they worked on in their own projects. That is, if they have any. Having any not just forked and click deploy projects in itself is a signal, that is worth looking into.
If all their previous employers don't allow side projects (must be in US or something, where employees don't have rights), then they should pay accordingly more to balance that restriction and loss in experience.
Which employers don't allow side projects? Not every side project has to be a SaaS hustle with Stripe billing.
My employer definitely doesn't own all the code I write in the evenings and on the weekends on my own time. Does yours?
We're gonna have to reinvent swe hiring
Expecting senior job applicants to have regained basic coding skills seems reasonable to me. I would be skeptical of an applicant who hadn't made the level of effort you're describing before applying.
A few days to brush up those skills really should not be too much to ask. I am against leetcode as much as anyone. But maybe just maybe refamiliarising yourself with well fundamentals is not bad idea. At least on the most prominent technologies you advertise.
The problem becomes distinguishing someone like you, who has the skill but hasn't recently used it vs someone who doesn't have the skill.
Leetcode was always a skill mostly practiced for interviews though, right? Arguably its a better signal now in the era of vibecoding that someone can do it themselves if they have to. It used to be "yeah of course I'm responsible in my job, I use a library for this stuff". But in this era, maybe performative leetcode has more value as a signal that you can really guide the AI.
Isn’t the solution to tell the interviewee that they will have to write some code without llm support? In the case of someone like you I’d hope they’d take that as notice to spend a tiny bit of time getting back up to speed. If it really is just a day or two then it shouldn’t be an issue
Yes, I fore warn all candidates that they will do a coding test, with examples of similar tests.
They are allowed to suggest the language they’re most familiar with, they’re told they don’t need to finish and they don’t need to be correct.
It’s just about seeing how they work through something.
If someone like the person you replied to would show up that unprepared , I would really question their own judgement of their abilities.
>> and I would get back on track in a few days
Thats the issue. How can one be sure you can actually get back on track - or - you never were on the track in the first place and you are just an AI slopper?
Thats why on interview you need to show skills. And on actual job you can use AI.
> then why am I paying for a senior ?
Because they know how to talk to the AI. That's literally the skill that differentiates seniors from juniors at this point. And a skill that you gain only by knowing about the problem space and having banged your head at it multiple times.
The actual skill is having knowledge and knowing when to not trust the AI, because it is hallucinating or bullshitting you. Having worked on enough projects to have a good idea about how things should be structured and what is extensible and what not. What is maintainable and what not. The list goes on and on. A junior rarely has these skills and especially not, when they rely on AI to figure things out.
>That's literally the skill that differentiates seniors from juniors at this point.
If your product has points where llms falter, this use a useless metric.
>and having banged your head at it multiple times.
And would someone who relied on an LLM be doing this?
Except most junior devs will be better than sr devs at wholehearted ai adoption
Grade inflation has spilled over into the corporate world. I’ve interviewed people titled “principal” who would barely qualify as “senior” a few decades ago.
"Senior" was already a weird title, given it could have been anything from 3-10 years of experience even back in 2021.
I've seen people with 10 years experience blindly duplicate C++ classes rather than subclass them, and when questioned they seemed to think the mere existence of `private:` access specifiers justified it. There were two full time developers including him, and no code review, so it's not like any of the access specifiers even did anything useful.
The jump from junior to senior means you can self start and have created enough of a network to seek out help. Junior used to be a 1-3 year training period. Senior to principal means you have signififcant positive impact across the company: upper management relies on you to define the roadmap. Most people hang out in ‘senior’ for their entire careers because they never have that drive to stand out. Thats why there are titles like “staff” and “senior staff” to promote people who don’t have what it takes to get to principal.
I always wondered why don't we have just developer level. That is for those with say 5-7 years of experience. That are skilled enough to do their bit on their own, but might not yet fully see the big picture on whole application scale for whole thing...
Straight from junior to senior just feels weird jump. Junior and Senior sound like adjectives to me. Qualifiers. And there should be some middle point in between where bulk of the workforce doing the actual job should be.
A lot of companies do have levels. See http://www.levels.fyi
There are specific cultures where titles and steady title progression are Really Important.
Me being from a place where they definitely aren't found this hilarious.
I've had meetings with Principal Architects with less experience than me (title: Backend Programmer).
Bigger organisations really should standardise their titles to specific experience/responsibility/capability milestones so people from other sides of the org can use the title to estimate the skill level of the other person they're talking with.
It's a matter of opinion what you should know and what you can easily google or ask an LLM.
If someone needs to continuously google how to use the basic data structures in a language they use every day, then I worry about their ability for knowledge retention as a whole.
“A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.”
― Robert A. Heinlein
(It's a matter of opinion)
I'm not sure I understand the point you're trying to make. Do you think a senior developer needs a basic understanding of the language they use or not?
I think you're approaching the quote the wrong way. It's more like "a C++ developer should be able to look at Javascript and ascertain the gist of what's happening, even if they can't finely dissect the exact issues on a language level."
Though the quote is more broad than that. It's really saying "a C++ developer should be able to write up a report, participate in an interview panel, plan a small party for Jane's birthday this Friday, and be able to make conversation with the external partner coming in today". Few of us are simply just writing code all day.
> Do you think a senior developer needs a basic understanding of the language they use
4 years ago, I'd have said "obviously".
At this point? Only for specialist languages the LLMs suck at. (Last I tried, "suck at" included going all the way from yacc etc. upwards when trying to make a custom language).
For most of us, what's now important is: Design patterns; architectures; knowing how to think of big-O consequences; and the ability to review what the AI farts out, which sounds like it should need an understanding of the programming language in question, but it's surprisingly straightforward to read code in languages you're not fmiliar with, and it gets even easier when your instruction to the AI includes making the code easy to review.
It’s not just the language but the domain within the language.
I see both the latest Claude and GPT models fall over on a lot of C++, Swift/ObjC and more complex Python code. They do better in codebases where there is maximal context with type hints in the local function. Rust does well, and it’s easier to know when it’s messed up since it won’t compile.
They also tend to clog up code bases with a cacophony of different coding paradigms.
A good developer would be able to see these right away, but someone who doesn’t understand the basics at the least will happily plug along not realizing the wake they’ve left.
Except I’m not hiring someone to do 20 things. I’m hiring them to do the one thing they say they can do.
Would I hire a taxi driver who can’t drive to drive me somewhere?
Why would I hire a software engineer who can’t demonstrate their abilities.
>I’m hiring them to do the one thing they say they can do.
If you hire an amazing programmer but then ask for a quick report on their implementation and it's gibberish. Is that satisfactory? Their job isn't to write reports.
If you ask them during lunch to grab your meal from the desk and they spill it everywhere, is that satisfactory? You didn't hire then for their ability to carry a plate.
EDIT: I should add here that I agree with your hiring approach. I just think you're critiquing this quote in the wrong way. The point of the quote is that humans have a capacity to learn and perform many tasks, even if they do some better than others. LLMs hinder this ability.
edit: wasting my time
Please re-read the original comment. All your suggestions are addressed there and I explain why this is inadequate.
I think it depends on what you consider "basic data structures"...
If it's the List and Dict (or whatever they are called in that language) then maybe. But I'd not expect someone to spell the correct API by heart to use Stack, Queue, Deque, etc, even they're text book examples of "basic" data structures.
For me, in a coding test, basic data structures are (using Python terms for brevity):
- lists
- dictionaries
- sets
- nested versions of the above
- strings (not exactly a data structure)
strings are tenuously in my data structures list because I let people treat them as ascii arrays to avoid most of the string footguns.
The question is: What are they better at? If you think the answer is "Nothing", I would be suspicious.
It’s never nothing, but it’s a matter of whether that something is enough of a differentiator.
More so, can they demonstrate that in interviews. We specifically structure our interviews so coding is only one part of them, and try and suss out other aspects of a candidate (problem solving, project leading, working with different disciplines) and then weight those against the needs of the role to avoid over indexing on any specific aspect. It also lets us see if there are other roles for them that might be better fits.
So a weak coder has opportunities to show other qualities. But generally someone interviewing for a coding heavy role who isn’t a strong coder tends to not sufficiently demonstrate those other qualities. Though of course there are exceptions.
Pair a senior with an agent LLM, pair a junior with an agent LLM, measure the output over some cycles. You will find your answer in the data, one way or another.
Truthfully, in my experience, they both end up performing near the level of the LLM. It’s an averaging factor not an uplifting one.
Right now we are seeing junior hiring fall off a cliff, whether or not LLMs are responsible is hard to say. But if it does turn out that everyone performs to the level of the LLM, then a vast amount of business dollars can be saved by hiring starving English majors exclusively and dispensing with STEM majors altogether.
I said it’s an averaging factor not an absolute match.
A senior with a LLM will likely still outperform a junior. A CS major with an LLM will out perform an English major.
But is the senior out performing the junior at a level that warrants the difference in salary?
Then people will point to the intangibles of experience, but will forget the intangibles of being fresh and new with regards to being malleable and gung-ho.
I keep hearing this however I was just at a career fair for University of Washington CSE new grads (representing my company) and it was packed with companies hiring. Obviously, this is just an anecdote, is there any good data showing the real situation?
It's anecdata from layoff reports and whatever people see from their own employer / industry.
To add my two cents of anecdotes from what I can tell in the creative agency world: shit sucks hard at the moment. Clients are cutting back budgets hard and a lot of them explicitly ask for AI strategies - obviously in terms of saving money. Blockchain craze was similarly bad when it comes to client requests for stuff that Just Did Not Make Sense, but at least there was money pouring in. And it's that way across the industry, everyone I know feels the same. There's no money from clients that would pay for juniors, so no juniors are getting hired even if it might blow up in five, six years when there are no fresh juniors/intermediates around.
Personally, I decided to part ways. There's many places I want to be, but the creative industry when the AI bubble and hiring anti-bubble finally pops? Oh hell no.
I mean, I don't know if I can specifically find new grad tech industry that gets a tech job. But definitely look at hiring numbers in tech and among the youngest demographic. They both aren't good, like pretty much all jobs right now outside of healthcare
The BLS works as well, for now. I think that's quickly going to change with future reports.
This was also making the rounds last ywar: https://futurism.com/the-byte/berkeley-professor-grads-job-m...
Nope, you don't pair senior dev with llm. You allow them use llm, but not an agent.
IF agent starts generating code - nobody will have time and stamina to rewrite all the slop, it will just get approved.
Copy/paste from chat is the only way to ensure proper quality so that developer can write high quality code and just outsource to AI tasks that are boring or generic.
> aren’t able to do junior level tasks without GenAI helping them
I’m assuming “unable” means not complete lack of knowledge how to approach it, but lack of detail knowledge. E.g. a junior likely remembers some $algorithm in detail (from all the recent grind), while a senior may no longer do so but only know that it exists, what properties it has (when to use, when to not use), and how to look it up.
If you don’t think of something regularly, memory of that fades away, becomes just a vague remembrance, and you eventually lose that knowledge - that’s just how we are.
However, consider that not doing junior-level tasks means it was unnecessary for the position and the position was about doing something else. It’s literally a matter of specialization and nomenclature mismatch: “junior” and “senior” are frequently not different levels of same skill set, but somewhat different skill sets. A simple test: if at your place you have juniors - check if they do the same tasks as seniors do, or if they’re doing something different.
Plus the title inflation - demand shifts and title-catching culture had messed up the nomenclature.
I don’t test rote algorithmic knowledge in our coding tests. Candidates can pick their language
Candidates can ask for help, and can google/llm as well if they can’t recall methods. I just do not allow them to post the whole problem in an LLM and need to see them solve through the problem themselves to see how they think and approach issues.
This therefore also requires they know the language that they picked to do simple tasks , including iterating iterables
That’s weird. Any senior developer worth their salt surely should know that LLMs produce a lot of weird nonsense with one-shot prompts, so they need to talk design first, then code the implementation.
This said, IMHO one-shot is worth a try because it’s typically cheap nowadays - but if it’s not good (or, for interview reasons, unavailable) any developer should have the skills to break the problem down and iterate on it, especially if all learning/memory-refreshing resources are so available. That’s the skill that every engineer should have.
I guess I must take my words back - if that’s how nowadays “seniors” are then I don’t know what’s going on. My only guess is that you must’ve met a bunch of scammers/pretenders who don’t know anything but trying to pass for a developer.
> including iterating iterables
I would've chosen a language without iterators, what would you do then??
It doesn’t need to explicitly be an iterable. But can you loop through an array as part of the problem solve.
That sounds really hard, you can only hope to solve that with AI. /s
Good luck, sounds like a fair hiring process.
Why not let them use LLMs? LLM's are a tool for the job so you wind the find the candidate that can most effectively use that tool in your role. If LLM's struggle with your technologies and API's then a developer that can use an LLM for development with good results should be a desirable thing, right?
Can the senior developer understand and internalize your codebase? Can they solve complex problems? If you're paying them to be a senior developer, it likely isn't worth their time to concern themselves with basic data structures when they are trying to solve more complex problems.
You literally asked the question the GP pre-emptively answered. Read their last paragraph.
>Can the senior developer understand and internalize your codebase?
Would you trust someone who needs llms in the hiring phase to be able to do this higher order tasks if they can't nail down the fundamentals?
What do you consider senior?
We've seem to have had a significant title-inflation in the last 5 years, and everybody seems to be at least a senior.
With no new junior positions opening up, I'm not even sure I blame them.
A senior to me is someone who can tackle complex problems without significant supervision , differentiated from a lead in that they still need guidance on what the overall tasks are. They should be familiar enough with their tech stacks that I can send them to meetings (after the requisite on-boarding time) to represent the team if needed (though I try and not overload my engineers with meetings) and answer feasibility questions for our projects. They don’t need constant check ins on their work. I should be able to bounce ideas of them readily on how to approach bigger problems.
A junior being someone who needs more granular direction or guidance. I’d only send them to meetings paired with a senior. They need close to daily check ins on their work. I include them in all the same things the seniors do for exposure, but do not expect the same level of technical strength at this time in their careers.
I try not to focus on years of experience necessarily, partly because I was supervising teams at large companies very early in my career.
Are you sure you're not just mostly seeing the candidates who are using LLMs to pass through the earlier screening phases with flying colors despite their lack of skills? (i.e., they haven't atrophied, they just weren't very good to begin with). There's always a lot of unqualified applicants to a job but LLMs can make them way more effort to filter out.
We have the coding test as the second phase now for specifically that reason, and might have more once they’re doing the full interview set.
That's what I said about compiled languages. No one knows how to optimize assembly anymore, they just let the compiler do it.
> Firstly, we’re making new technologies and APIs that LLMs really struggle
LLMs absolutely excel at this task.
Source: Me, been doing it since early July with Gemini Pro 2.5 and Claude Opus.
So good, in fact, that I have no plans to hire software engineers in the future. (I have hired many over my 25 years developing software.)
So you’ve made a decision based on three months of use.
I am legitimately interested in your experience though. What are you creating where you can see the results in that time frame to make entire business decisions like that?
I would really like to see those kinds of productivity gains myself.
I'll give you an easy example. LMDB has a (very) long-standing bug where "DUPSORT" databases become corrupted over time.
Separately, I wanted to make some changes to LMDB but the code is so opaque that it's hard to do anything with it (safely).
So I gave the entire codebase to Gemini Pro 2.5 and had it develop a glossary for local variable renames and for structure member renames. I then hand-renamed all of the structures (using my IDE's struct member refactoring tools). Then I gave the local variable glossary and each function to Gemini and had it rewrite the code. Finally, I had a separate Gemini Pro 2.5 context and a Claude Opus context validate that the new code was LOGICALLY IDENTICAL the previous code (i.e. that only local variables were renamed, and that the renaming was consistent).
Most of the time, GPro did the rewrite correctly the first time, but other times, it took 3-4 passes before GPro and Opus agreed. Each time, I simply pasted the feedback from one of the LLMs back into the original context and told it to fix it.
The largest function done this way was ~560 LOC.
Anyway, the entire process took around a day.
However, at one point, GPro reported: "Hey, this code is logically identical BUT THERE IS A SERIOUS BUG." Turns out, it had found the cause of the DUPSORT corruption, without any prompting—all because the code was much cleaner than it was at the start of the day.
That is wild to me! (It actually found another, less important bug too.)
Without LLMs, I would have never even attempted this kind of refactoring. And I certainly wouldn't pay a software engineer to do it.
> What are you creating where you can see the results in that time frame to make entire business decisions like that?
I've developed new libraries (using GPro) that have APIs it can reliably use. What's easy for LLMs can be hard for humans, and what's easy for humans can be hard for LLMs. If you want to use LLMs for coding, it pays to need code they are really good at writing. AI-first development is the big win here.
(This is all in Clojure BTW, which isn't really trained for by vendors. The idea that LLMs are ONLY good at, e.g. Python, is absurd.)
While your lldb use is interesting,, I’m very much interested in what you said you’re doing greenfield. What is the scope of that?
Clojure has enough code out there that it’s well covered by the major LLMs.
What's the bug ID, have you submitted the fix back?
That was my plan, but I haven't yet--the fix was in my renamed code and I haven't put in the work to make it correspond to the original code. But here's the commit message, maybe you can see it easily:
And here's the function with renames (and the fix): I have confirmed the fixed logic seems correct, but I haven't written a test for it (I moved on to another project immediately after and haven't returned back to this one). That said…I'm almost certain I have run into this bug in production on a large (1TB) database that used DUPSORT heavily. It's kinda hard to trigger.Also, thanks for a great library! LMDB is fantastic.
Thanks for that. It doesn't look like there is an open bug report for this yet.
I understand your commit description but I'm still a bit puzzled by it. cursor_to_update shouldn't have a deeper stack than cursor, even if it was a cursor on a subdatabase. All the references to the subdatabase are in the subcursor, and that has no bearing on the main cursor's stack.
The original code with the bug was added for ITS#8406, commit 37081325f7356587c5e6ce4c1f36c3b303fa718c on 2016-04-18. Definitely a rare occurrence since it's uncommon to write from multiple cursors on the same DB. Fixed now in ITS#10396.
> Fixed now in ITS#10396.
Thanks, appreciated.