I don't think those explanations are mutually exclusive.
Yes, there's a large cohort of "senior" software engineers who can't actually code. They bullshit their way into jobs until they're fired and then apply for the next one. These are the people you want to filter out with live coding.
But also, someone can fail a live coding interview for reasons other than belonging to that group.
I think there's a lot of developers who can ace a live-coding interview but who lack the understanding of engineering systems at scale so they'll make your whole codebase worse over time by introducing tech debt, anti-patterns, and inconsistencies. These are the people you really want to avoid, but very few interview processes are set up to filter them out.
There's an assumption that the company's existing senior architects and developers will stop a new person from making the code worse, but also devs at every company thinks their codebase is terrible so it obviously isn't working.
I've seen lots of devs who think their codebase is the only correct way to do things. Lots of overconfident people out there. Inconsistencies are fine as long as there's file level consistency. All that really matters is if you can relatively quickly understand what you are working with. What you really want to avoid is having functions doing 20 different things from 5 different contexts.
Exactly. One negative productivity technical tornado can cause more damage than 10 people who lied about their coding ability.
What engineering interviewing process catches things like this?
Most coding interviews are accompanied by work experience discussions which CAN identify stuff like this, although obviously people can BS.
Work experience discussions are the best I’ve come up with.
What I’m suggesting is hire experienced people based on that and resume verification and behavior interviews like nearly every other job on the planet.
And if someone lies about their ability to actually code, fire them quickly.
I agree. Live coding always has a much smaller scope than real software, and after a few interviews it is easy to learn to read the room, even for the worst developers.
I think we can leave companies who don't care about quality out of the discussion, but for those who do, the time to detect those developers is in a probational period, which is not something that most companies really use on their favor.
The problem is this requires a good management that is able to actively paying attention to the work of the developer. Which is often not in place, even in companies who want to prioritize quality :/
Have you interviewed people recently? The "worst developers" absolutely cannot solve basic problems.
Yes, I agree. Good point.
A fizz buzz or something similar is often already too much for these.
You could filter then out much more effectively by letting them sit in a room by themself to write the code, that way you aren't missing out on good candidates who can't function when under abnormal stress(that has nothing in common with the actual job).
I've had take home problems for job interviews that were given a few days before and during the actual interview I only had to explain my code. But I wouldn't be sure this still works as a useful candidate filter today, given how much coding agents have advanced. In fact, if you were a sr dev and had a bunch of guys to bounce this problem back and forth, it wouldn't even have filtered out the bad ones back in the old days. There is very little that is more telling than seeing a person work out a problem live, even if that sucks for smart people who can't handle stress.
I have found over the years that I learn more by asking easier questions and interacting with candidates as they think through problems out loud. Little things that test the critical ability to craft a Boolean expression to accurately characterize a situation can be explored in a brief interview where you have some assurance that they're working on their own, and not just getting an answer online or from a smart roommate. (Sample: given two intervals [a,b] and [c,d], write an expression that determines whether they intersect.). Candidates that have lots of trouble programming "in the small" are going to have trouble programming in the large as well.
What I find effective (on both sides of the interview table) is not only asking easier questions, but active encouragement to first talk out and work through the most fundamental, basic aspects of the problem and it's simplest solutions before moving into more advanced stuff.
I think a lot of experienced people's brains lock up in an interview when asked simple questions because they assume that they're expected to skip straight past the "obvious" solution and go straight for some crazy algorithm or explain the fine points of memory/time tradeoff. This often doesn't present as intended - it looks to the interviewer like you don't even know the basics and are grasping at straws trying to sound smart.
If people can explain their decisions, I'd say it's fair game. It would be nice to know up front if someone used AI of course.
The other implication here is that if a candidate can use AI for a take home and ace the interview, then maybe the company doesn't have as tough of problems as it thought and it could fill this seat quickly. Not a bad problem to have.
Leetcode for CRUD app positions is overkill.
I don’t use those LLM tools, but if someone can pass the test with LLM tools, then they can pass the test unless there’s something special about the environment that precludes the LLM tools they use.
Even before the coding assistants I was not happy with homework. It was not discriminatory, everyone looked the same, https://andrewpwheeler.com/2022/03/24/i-have-no-clue-how-to-....
> But I wouldn't be sure this still works as a useful candidate filter today, given how much coding agents have advanced.
Prior to ChatGPT coming out, I gave a take home test to sort roman numerals.
What before was a "here's a take home that you can do in an hour and I can check the 'did you write the code reasonably?'" is now a 30 seconds in ChatGPT with comments embedded in it that would make an "explain how this function works" to be less useful. https://chatgpt.com/share/688cd543-e9f0-8011-bb79-bd7ac73b3f...
When next there's an interview for a programmer, strongly suggest that it be in person with a whiteboard instead to mitigate the risks of North Korean IT workers and developers who are reliant on an LLM for each task.
My solution for this was a propose a problem obscure enough that no LLM tool really knows how to deal with it. This involved some old Fortran code and obscure Fortran file format.
How would someone explain code that was vibe-coded?
You can have the AI explain it to you. There's also a middle ground between vibe coding and "I can code some things but never could have coded this without an AI".
Doesn't even have to be AI. Give me some random file from the Linux kernel and I could probably explain everything to you if you gave me a few hours. But that doesn't mean I would ever be able to write that code.
I don’t disagree, but in those interviews the explanation is also a bit of a q&a, so an effective interviewer can detect candidates who only memorized things. Someone who can ace a q&a about Linux code is already better than average.
Isn't that a good thing? The fact that the candidate dumped out code that they didn't write is often called "cheating". The fact that candidates can't explain it (because they didn't write it) means it's a good test of something most interviewers find unacceptable.
Are you asking how they would get that info they didn't have / couldn't come up with? Because you can literally have a chatbot explain every line to you and why it is there and even ask it about things you don't know like a teacher. And for simple problems this will probably work better than with actual humans.
I assume questions to explain the code would be extremely specific about why you did something on a specific line, or why you chose to design this part that way instead of another, to detect plagiarism and vibe coding, not a request for a prepared monologue.
We live in a remote world where a hiring process like this is less of an option in most cases
Leaving aside that many companies have pulled back from remote to at least some degree, I'd always push for an in-person day for a variety of reasons. In general, the cost is nothing for a late-stage/end-stage confirmation. And, honestly, a candidate that just doesn't want to do that is a red flag.
> In general, the cost is nothing for a late-stage/end-stage confirmation.
One in-person day costs a nearby candidate about 3 days, and a more remote candidate anything from a week to a couple of months depending mostly on where you are. And yeah, it also costs some money that maybe you will reimburse.
It doesn't cost you much, that's for sure. But if it's for a full-remote position, it's absolutely not a "the cost is nothing" situation and the candidate refusing it for some random company in a random stage of the interview is absolutely reasonable.
While I don’t disagree with you I find it to be a slippery slope to some extent.
Would you screen out Linus Torvalds because he hypothetically doesn’t want to come in to a physical office for an interview?
Hiring managers should think long and hard in a data-driven way about whether the office presence is so necessary that you are willing to miss out on the best candidates who have the luxury of being picky.
Is it true scientifically that an in-person interview day results in better candidate quality or is that just a vibe?
I think eliminating top talent who refuse to step foot in an office and are rare enough to be able to maintain that demand is a lot of quality people being left out of your talent pool. I thought during the pandemic we already proved by numerous studies that in-office workers are less productive.
My company philosophy would be more like, put the burden of identifying quality talent on the employer rather than the employee. Put the candidate through the minimum effort required to screen them and identify standout talent. Then when you find that standout talent you roll out the red carpet and focus on convincing them to work at your company.
You can come up with outlier examples of course--though I'm not sure how relevant they are unless you're looking at hiring a "name" for some reason. But I'd still default to an in-person visit of some sort. I've never seen any data but then in-person was just assumed in most cases until a few years ago.
Read your last two sentences over again. That’s exactly my point: it’s all an old habit that isn’t based on outcomes.
I think it’s a human social instincts thing and not a quantitative thing. There might be a better way but we default to the social ritual because ape brain is most comfortable with it.
I don't know about that. Long ago I interviewed with someone that wanted some trivial C++ thing written on their laptop. I hadn't seen a Windows dev machine before and had no Internet access. I think I'd worked out the compiler was called visual studio and how to compile hello world by the time limit. Not sure that told either of us much.
I share your point of view, but live coding these days are just beyond that testing programming skills. You must know by heart the most common algorithms out there and design solutions that might involve two or three of them to solve a problem in 30 minutes.
Sometimes you spend the whole time trying to figure out how to solve the puzzle that don't even have time to show that you can - actually - code.
> You must know by heart the most common algorithms out there and design solutions that might involve two or three of them to solve a problem in 30 minutes.
You're not going to pass every interview. Some of them are truly outlandish, and require all of the above.
What you need is the technical knowledge to translate requirements into a loose pattern like "This looks like a search problem", then have the charisma (or more accurately, practice) to walk the interviewer through how each search algorithm you know could apply there. Then of course be able to actually write some code.
I've passed interviews where I had never heard of the datastructure they wanted me to solve it with; I just walked them through the tradeoffs of every data structure that I knew applied to it instead.
I’ve been doing this for 20 years. I’ve never worked with a single one of those people. I don’t think I’ve ever even interviewed one where I couldn’t have screened them out based on their resume and a 15 minute conversation.
I’ve worked with plenty of people who passed a whiteboard interview and then spent years actively reducing aggregate productivity.
Why do you need arbitrary (and very short) deadlines, and for someone to stand up at a whiteboard while simultaneously trying to solve a problem and "walk you through their thought process" to filter out people who can't write code on the job?
The short deadlines are because neither the company nor the candidate wants to spend a month on an extended interview. Solving a problem and walking through the thought process are because that's what "coding" is.
> neither the company nor the candidate wants to spend a month on an extended interview.
So says the companies that insist on multi-round, multi-week interview loops.
I don't know about you, but I've never had to live code a PR and explain to my reviewer what I was thinking while writing the code. By "deadlines" I'm referring to the length of the interview. Take home problems theoretically solve both these issues, but they need to be properly scoped and specified to be valid assessments.
I sit down with juniors and sketch out designs or code for them while talking through the thought process at least once a week, and even when solo coding, I expect everyone produces work that explains itself. For particularly complex/nuanced changes, people do hop on a call to talk through it.
Like I said the deadlines work for both sides. If a company wants to give homework instead of having their own senior engineers spend time talking to me, that tells me what I need to know about how they value my time.
> I sit down with juniors and sketch out designs or code for them while talking through the thought process at least once a week, and even when solo coding, I expect everyone produces work that explains itself. For particularly complex/nuanced changes, people do hop on a call to talk through it.
That's not equivalent to what I said, nor is it live coding.
Again, those deadlines are artificially short compared to real world scenarios, and completely arbitrary. They are so short, in fact, that they render the interview an invalid test of real working ability. A work sample has been proven time and again to be the most valid measure of whether a candidate can actually perform the job, but the conditions under which a live coded "work sample" is performed in an interview render it invalid.
It's not artificial: the company has a day of my time. I have a day of their time. We both want me to meet several people on the team to see if it's a good fit. Because of the constraint, we keep it to relatively simple discussions around toy problems that can be solved in an hour.
Yes, it is artificial. Everything about a live coding interview is artificial. Code doesn't get written in 1 hour blocks while someone's watching over one's shoulder, all the while asking questions to interrupt one's thought process, in any company I've ever worked for.
Like I said, this is literally a thing I do all the time. I have standing 1 hour blocks for each of my team members every week and it's not uncommon for us to build out the skeleton of a problem solution together. I literally did what you said on Wednesday for someone for a gitlab change because I don't expect they know how secret injection works, but I want them to know. And absolutely I've encouraged them to ask questions, and I ask them questions to check their understanding.
Like I said, that's not the same. It's not comparable in terms of stakes, or expected outputs. You're not addressing the issue.
[dead]
In most of the western world, firing employees is a high risk, high cost task. Ideally companies would hire quickly and fire poor matches just as quickly once they've been evaluated in the real world environment of the company. For this to work, on the employee side there needs to be knowledge that this is the company's process, financial depth to deal with the job not being stable, and a savviness to not relocate into a job that's risky. On the employer side, there needs to be a legal and social environment that doesn't punish removing non-productive employees.
The legal environment is what it is and unlikely to change. The social environment is fickle and trend driven. Workers can't always fully evaluate their odds of success or the entirety of risk of leaving a job that's valuable for the employee and employer for one that might end up as a poor match, even if both sides have been transparent and honest. It's a difficult matchmaking problem with lots of external factors imposed and short term thinking on all sides.
Ideally young workers would have an early career period that involves a small number of short lived jobs, followed up by a good match that lasts decades, providing value to both the employee and employer. Much like finding a spouse used to be a period of dating followed by making a choice and sticking with it so a life could be built together, employment ideally should result in both sides making the other better. Today however everyone seems focused on maximizing an assortment of short term gains in search for the best local timescale deal at the expense of the long term. It's interesting how the broken job market and broken family formation process in the western world mirror each other so much.
I suspect the similarity is just a coincidence.
There was a lot of social pressure in the past for permanent marriages. That doesn't mean they were happy marriages. With the social changes in the west in the 1960s, divorce became more socially acceptable. Legal changes meant women had the ability to join the workforce and support themselves. People in unhappy marriages had options to seek happiness elsewhere. Those options didn't exist before.
For job retention, the problem is that changing jobs is often the only way to advance. I lost my best worker because the suits wouldn't give him a raise. He now makes more than I do at a different company. He liked his job with us, but he tripled his pay by leaving. My coworkers all tell the same story. I'm one of the lucky ones that managed to move up in the company, and that's only because I had the boss over a barrel.
There are two ways to interview:
1. Make sure you pick every good candidate, but some bad candidates will slip through as well.
2. Make sure you reject every bad candidate, but some good candidates will fail as well.
Candidates want process #1, but companies have no reason to push for it. The cost of accidentally hiring a bad employee is simply too high, way more than rejecting a good employee. The current system in place prioritizes #2. Yes they are rejecting great candidates, and they are aware of it.
The article is suggesting that #2 will end up rejecting LOTS of good candidates (and potentially ALL female candidates)
Also I don't think being artificially picky is a better filter than just going with some gut feeling after weeding out candidates with fake credentials.
Being picky gives the illusion of choosing when you in practice are bound down by the process.
> Yes, there's a large cohort of "senior" software engineers who can't actually code. They bullshit their way into jobs until they're fired and then apply for the next one. These are the people you want to filter out with live coding.
Genuinely, are there any amount of these at any significant scale in a place like Silicon Valley? I'm not sure I've ever met someone who couldn't code at any of the places I've worked.
Senior engineers are heavily evaluated for their ability to pump out code. If you're not coding, what the hell are you doing at a startup that needs features built to generate any revenue?