I recently did a round of interviews at various AI companies, including model labs, coding assistants, and data vendors. My first takeaway is that, wow! the interviews are very hard, and the bar is high. Second, these companies are all selecting for the top 0.1% of some metric - but they use different metrics. For example, the coding assistant interview focused on writing (what I felt was) an insane volume of code in a short period of time. I did not do well. By contrast, another company asked me to spend a day working on a particular niche optimization problem; that was the entire interview loop. I happened to stumble on some neat idea, and therefore did well, but I don't think I could reliably repeat that performance.
To reiterate - wow! the interviews are hard, every company is selecting for the top of a different metric, and there's really no shame in not passing one of these loops. Also, none of these companies will actually give you your purpose in life, your dream job will not make you whole:-)
My career long experience with these types of interviews is you get hired by the company that, when they interview you, you get lucky and they happen to ask the questions you’ve just brushed up on or you get lucky and see the answer quickly for some reason. The content of the actual work I’ve done at these companies and how the work is done, is completely different to these interviews and I’d have done equally well at all the places that didn’t hire me because they happened to ask the wrong questions.
I know, because I’ve been rejected and accepted to the same company before based on different interview questions, and did just fine in the role once I was in there.
In short, if you have decent skills the tech interviewers are mostly total random luck IMO, so just do a bunch of em and you’ll get lucky somewhere. It won’t make any rational sense at all later where you end up, but who cares.
>you get lucky and they happen to ask the questions you’ve just brushed up on or you get lucky and see the answer quickly for some reason
My experience exactly! I've been lucky in most of my interviews that I was asked about things I just happened to brush up on or had thought about deeply in some past project, so I was offered the job.
And like you say, the job rarely demanded any of the things I was asked about... which worked against me once, where I sailed through the interview process but struggled for the first year to get up to speed in my actual day-to-day job, although I did manage to get my act together before it became a big problem.
Yup! To be honest, it should be obvious via someone’s resume and references whether they’ll be capable of adapting to the job without making them jump through random hoops. People treat hiring devs like they’re hiring a contractor to paint their bathroom, instead of hiring them like white collar professionals who will grow with a position.
Arrrgh I remember an interview where I got this lucky... and I ended up failing it miserably. It was a python-heavy position, and I had been watching some Peter Norvig videos in the weeks beforehand to prepare. They asked me to implement some basic functionality of a poker game, which was EXACTLY what one of the videos was about. I was trying so hard not to copy his approach, and my own 'natural' approach would have been fairly similar (but not nearly as elegant), so by trying to avoid both of those approaches I made a complete mess, haha
And unfortunately, from the point of view of the company, this is a feature not a bug:
* to the company the cost of a false positive (bad hire) is very very much higher than the cost of a false negative (passing on a good candidate).
* AI companies have a large pool of strong candidates to interview
* Therefore they are incentivized to make their interview process hard enough that a poor candidate almost never passes it
* but then it becomes something a strong candidate can only pass with a bit of good luck
This is not “fair”, but it’s a marketplace. The best approach is the one you propose: accept it and don’t take it personally if you miss, roll the dice again.
Why is it that most other jobs especially low skill take the opposite approach? You screw up or demonstrate your incompetence on your first day on a construction job site you are let go right then.
I think it's trickier to gauge in knowledge work because there's a ramp-up period, even for top performers. Just understanding the institutional context that led to the current ecosystem - essentially understanding every Chesterton's Fence you encounter - takes a substantial amount of time.
Typically there's a lot of onboarding, and even a good candidate might not get a lot done in the first month or three... By the time you realize the new hire isn't a good fit, you've spent a ton of time on training.
Otoh, if you hire me to frame a house, it'll be objectively clear you need to get rid of me in the first hour, if not the first ten minutes. I don't know how I'd get past a screening for that either, but still.
They don’t have the money to hire lawyers to pursue discrimination suits, and the potential SME employers frequently don’t have the balance sheet to be collectible.
It’s a different ballgame when you can gamble $20k to make $1M.
They just have a pool that’s filled with bad candidates. They want to disable luck for them.
Ironically, they turn on luck even more.
No it isn't, their strategy is great at increasing the rate at which they select those deadly "bad hires". There's just an insane amount of risk in doing these sorts of tech interview things; code up a quick monte carlo simulation to convince yourself if you like. It's just that the risk doesn't fall on the improperly aligned humans conducting the interview, it's offloaded onto the company.
Exactly, you could just as easily hire a bad candidate because they got lucky, and this happens all the time.
I mean, they could just as easily hire a bad candidate that happened to cram that specific interview knowledge the day before but is otherwise a bad developer.
Personally I don’t agree with the crazy tech interviews at all, no other job on the planet does it like this, even high skill jobs like doctors or professors. They hire based on your experience, your references, and a good chat to make sure you’d be a culture fit. If there’s a serious problem with a doctor once they start the job, they’d be let go, but they’re professionals so this isn’t a super common occurrence.
The exact same process could be taken with developers IMO. If I have a seasoned career, you should hire me based on that, not because I jumped through hoops you set up.
Like the great Mike Tyson once said, "God punishes you by giving you everything you want... to see if you can handle it".
For many, achieving your dreams usually comes with the hard lesson that you had the wrong dreams and that the real dreams you should have had were many of the things you already gave away to get there.
Then again, infinite AI-developer money isn't the worst outcome, either. Something something land among the clouds.
Honestly, AI is far too creative to pick a job that cares about the wrong thing and you are dumped into a feature mill. It’d be the worst time to be at these large companies because the smaller more creative startups are what’s going to be really an adventure in this space, versus the same old run of the mill career treadmill SWE have been given after SWE ballooned in the 2010s.
You have much cooler opportunity in these new companies and product spaces versus the large ships. It really is a once in a lifetime opportunity.
> the coding assistant interview focused on writing (what I felt was) an insane volume of code in a short period of time.
I had a bunch of these my last round of interviews, and am not convinced most companies even know what they want from these or how to assess. In the majority of cases it was clear to me the interviewers never even read the code I had submitted.
As an example, one company wanted a full AI Question/Answer system for large code projects using RAG to work on an arbitrarily large code base with an eval suite to go with it and also an API endpoint that could be called to automate asking of questions. You only had 24 hours to complete the assignment from receipt.
It was clear even before this that the company was likely not a good match, but I wanted to implement it for fun anyway. I'd already built all of this in production at a previous job (though it took weeks, which still felt pretty fast) so I knew how to approach it. Got it all done in time, met all acceptance criteria, had it so the entire thing could be run with one line of code (including building the RAG system, running evaluations, starting the webserver for the running API endpoint)... and rejected with no feedback a week later.
The trouble with asking people to write massive amounts of code in short periods of time is that you actually have to review it. I also have to say, despite the competitiveness, I was pretty unimpressed with the technical skills of the people on these teams (mostly smaller AI startups for me). It takes a pretty skilled engineer to assess the quality of a code base in such a short period of time and these teams did not seem like that had a lot of extra time on their hands.
Did you get offended when they rejected your day of work without even commenting? I guess they could've at least flattered you a bit saying it was impressive, but they had to go with another candidate for reasons.
I know I'd be pissed if the work did go uncommented.
I tend to reject full on large homework assignments like that unless I get a guarantee of feedback.
> Got it all done in time, met all acceptance criteria, had it so the entire thing could be run with one line of code (including building the RAG system, running evaluations, starting the webserver for the running API endpoint)... and rejected with no feedback a week later.
that is absolutely insane.
Are you looking? Can I hire you?
> Also, none of these companies will actually give you your purpose in life, your dream job will not make you whole:-)
Some people really do find a whole lot of personal meaning from their work. And that's okay. It's their life.
If someone is the sort of person who might find meaning working for Anthropic, they would find that meaning at a lot of other jobs as well. I think that's a better emssage; not that "you shall not find purpose in your work", but "the purpose you may find from work is not limited to a single or even small number of AI companies".
That is fair. I suppose what I meant is, the idea of working at one of these companies can be really exciting, almost a fantasy, but in practice: it might actually hurt you in many ways. 'Look what they make you give', as a certain character once said. With that said, obviously I think it's cool and worth doing, but there are significant and painful downsides, too.
If the past 25 years of tech companies is any indication of the future of these new AI endeavors, working there will be directly contributing to the enslavement of mankind in ways we can't even begin to imagine yet.
The greenfield projects arising from this leap look benign now, but I can almost guarantee that won't be the case in the next decade once these technologies optimize their revenue generation engines and enshittification takes hold. Humanity will be at the whim of the AI compute overlords much more so than we are now, and that's an inevitable nightmare dystopia that I'm not looking forward to. The gilded age will look like child's play by the time we figure this out as a society.
I suppose that if your ambition is to be on the winning end of that hellscape, then by all means, go for it.
I mean, if there's going to end up being a hellscape it'd be better to be on the winning end than the losing one, right?
I get it, we all want to be somebody, but is the juice worth the squeeze?
They may find a candidate that succeeds, they may not. In the end, it’s up to you to decide whether that kind of environment is for you. I also interviewed at a few AI startups and while difficult, I wasn’t impressed with them. They seem to be too high maintenance with little to no experience.
This is key: the OP seems to be putting them on a pedestal, but if Sturgeon's law holds (and I think it does) then a sizable percentage of what's happening there doesn't smell very good.
I totally agree. My work is a major part of my life. I do my job well and that's important to me.
But I don't care about any particular company. I'm just as happy automating refineries as I am factories or chemical plants. I just want to be a valuable member of a team that gets stuff done.
The blue-collar version of this, which I think distills the essence well: "Does life start when you clock in, or clock out?"
Very critical difference in mindset and the reason a lot of these conversations end up talking past each other.
> To reiterate - wow! the interviews are hard, every company is selecting for the top of a different metric, and there's really no shame in not passing one of these loops. Also, none of these companies will actually give you your purpose in life, your dream job will not make you whole:-)
I wholeheartedly agree.
That said, if you pursue jobs that are at least somewhat aligned with your personal convictions and desire for impact on the world and not strictly based on money, prestige, and power, you might be able to able to derive some greater sense of wholeness through the work. Not "make you whole", but "contribute to your wholeness".
If you only chase money, prestige and power - which if we're honest is central to a lot of the tech industry today - you probably won't experience any wholeness contribution from your work. That's fine, of course. Hopefully in that case you can find it through family, friendships, or community.
There's people whose entire job is to list houses at 50% over the market pricing. And there's people whose entire job it is to offer 50% market price for houses. They only need to be successful once every couple of month. They flood the market with buying and selling signals, that's just the way the market works. I wouldn't be discouraged if my fair market offer were rejected by most bidders in a market, statistically they are professional over/under bidders, that's fine.
They are all roundabout ways of getting to IQ. It used to be Leetcode but since that has become a known quantity and quite gamified, the interviews have evolved. Most of all, I have seen that there is crazy focus on speed and volume. Do a lot, read a lot of code, in little time. They've also adapted for AI coding tools. So the interviews have a debugging component, or take homes are open web, open tools, but again really compressed timelines.
The last bit - don’t tell people. Statistically most won’t reach their end goals, the few that do have reasonable chances of mental breakdowns and or other mental anguishes since - as the philosphers nailed a long time ago - the journey is actually what matters, the destination isn’t.
(Got where I wanted, the fact that I didn’t want enough and could re-set goals helped but still not great to reach the self imposed dream goal.)
Hi, can you share more about the coding assistant interview? Was it a task you had to solve and you were allowed to LLM to speed things up?
Oh, what I meant was, this was an interview with a company that produces a coding assistant. Ironically, you really weren't supposed to use an LLM much for the interview, although I think you could use the company's product if you wanted. The restrictions were a bit unclear to me and I chose to try to code everything by hand, but in retrospect, that was probably not feasible.
The assignment was hard for me for a few reasons: There were many requirements; the requirements as stated were quite unclear to me; and it turned out that they were really referencing a certain standard solution to a standard problem that I was unfamiliar with. So we spent half the interview explaining the idea to me (bad!), and then the other half was me trying to code it up, and I just didn't get nearly far enough.
Thank you. Interesting. Can you share a mock example of the problem you had to solve?
I think a spiritually similar assignment would be: implement multiplayer Google Docs using a CRDT, server, and clients, supporting a certain class of documents that are shown to you during the interview.
And imagine that the CRDTs are not directly explained in the problem statement, but some supporting data structure of CRDTs is referenced, and like clearly there is some way to make all the pieces referenced work together to create a working system, but it's never made explicit.
And imagine that you need to actually implement CRDTs from scratch in the interview, but also integrate them into this client-server document-editing app, which you also need to code from scratch. All in an hour.
This is why I say the interview felt kind of insane to me. But plausibly if I'd previously studied that problem, so that the implementation details were all clear to me from the start, and if I had used the company's coding assistant to the extent allowed, and if I had been in much better coding-things-from-scratch form, I could have done OK. I don't know. It's fine. They were probably looking for a type of engineer that I'm not. :')
This sounds like a take home assignment. Are these type of questions standard interview question today?
Nope, it was a live coding interview, less than an hour. I personally did a double-take at the beginning of the interview. However, the long tail of people who can code up algorithms and apps quickly is very long indeed. There are certainly people I've worked with who are just multiples faster than me in this kind of situation, and I could imagine them passing the interview.
And yet all of these companies' frontend is just plain ridiculous.