Take home assessments can be a valuable part of an interview process but they absolutely need a time limit. I think 2-3 hours is going to give you all the information you need, unless what you're selecting for is new grads with no dependents, hobbies, or responsibilities.
If this had been limited to 3 hours then the worst case is that the candidate would have lost 3 hours, but far more likely is that they would have come up with an entirely different proposal and/or solution that was appropriate for that timeline, and that extra information would have made it clearer what the company was looking for.
The other point I'd always encourage applicants to confirm is: are you looking for any answer, or are you looking for a good answer. Some take-home tests are purely about passing a test suite and how you manage it doesn't matter, some would prefer that you meet 80% of the requirements but write better code. I've seen applicants do the wrong thing on both sides of this.
To me, as someone who hires and gives candidates take home assignments, Kagi’s assignment is huge and disrespectful of the candidate’s time. Surely they’re (probably unintentionally) filtering for candidates that have a lot of time to waste on projects like this — while people who are busy (the kind of people you want) will pass.
Surely there must be other ways to have an idea on a candidate’s skills.
In our case, hiring people with data engineering skills, we just ask of them a simple ETL challenge (pull data from zip file, transform it, insert into any database). We leave ambiguities in there and there are some Easter eggs in the dataset (eg null values where you would not expect it, incorrectly formatted CSV) that we use to evaluate how well a candidate can perform.
We timebox it at 4 hours, but don’t give guidance in case they run out of time, that’s a good suggestion.
During a follow up call, we review the code together and we’ll ask them questions on how to improve the code (“what if the dataset doesn’t fit in memory”, etc), which is what the actual technical assessment is. At that point they should already feel somewhat comfortable with the problem domain, and you can assess their real skills.
I agree about Kagi, but about your process wanted to ask: do you really feel like having them write the code as a take-home task adds any signal to the process?
At my current gig we do a 1-hour pair programming kind of thing, where we video-chat and watch the candidate work on a small, straightforward task. And as the interviewer I watch how they use the tools, where they go for docs, do they read the requirements, how do they test, etc. By the end I always feel like I have a strong picture of their ability, and the whole thing is capped at 90 minutes for both sides (adding time for their questions).
If the candidate's code was written offline beforehand, I'd have no idea whether it was theirs, whether it came from a friend or chatGPT, whether it took ten minutes or ten hours, etc. Sure I could try to suss out those things during discussion, but isn't it better to observe directly?
Leaving aside the non-trivial question of candidate nervousness in live coding, I do find the "ask questions about take-home" process very illuminating.
If it came from a friend or chatGPT, you'll learn that quickly because you can ask them stuff like "show me how you're implementing FOO. OK, if the requirements change, and now we need to BAR, what do you need to change across the app?" If they wrote the code and understood the task, they should understand what needs to change, how to do it, etc, and you can do that part with them live. Maybe someday LLMs will be fast enough that you can't catch someone constantly waiting for the AI to help them, at least as of May 2025 we're not there yet.
This is a data engineer’s fizzbuzz and you would be surprised at how many people actually fail it.
If they didn’t write it themselves, they will absolutely fail in the follow-up session where the real evaluation happens and I ask them to scale the code, handle crash safety without using transactions, etc.
Some people are nervous during live coding challenges, which is why we decided to allow this to be taken home, and the candidate becoming familiar with the problem domain in their own time, rather than "live" under pressure.
Would you think doing this live in a screen sharing session adds more value? How would you prepare the candidate for this?
> Would you think doing this live in a screen sharing session adds more value?
In my experience, hugely yes. Above all else I really like that the time commitment is fixed on both sides - no homework for the candidate, and we're not asking them to invest any more time than we are.
Also doing things interactively gives a lot of leeway for adjusting and avoiding wasted time. If somebody's got one section basically solved, I might cut in and say "yeah that bit is great, let's call it finished and look at this other bit". If they're obviously having a lot of trouble, I might ask about their current work and maybe switch tasks if there's a better fit. Or if they're obviously coasting easily I might cut in with "how would you scale this?" questions that give more signal than watching them finish up the loose ends.
Another thing I like is that we can keep the task specification simple, like what you'd get from a PM, and it's up to the candidate whether they want to ask questions or jump in. I imagine that with take-home tasks you either need to give out pretty specific requirements, or else have people interpreting the task a variety of different ways.
> How would you prepare the candidate for this?
We explain at the beginning that the goal is write code for an hour and watch how they work, and there's no hard requirement for what needs to be finished by the end. And they're welcome to search the web, use AI, or anything else they'd typically do while working.
And I've only done a few dozen of these, but personally I've never seen anybody get particularly nervous. I think nerves are a bigger issue with HR style "pass all the tests within the time limit" tests, where the danger is that somebody can hit a wall and never get past it. That doesn't happen with my format, because if somebody is stuck I just give them hints until they're unstuck (since I get no signal from watching somebody scratch their head).
But if you're unable to determine if someone has this knowledge via a phone call, instead of 4 hours of their work without you even being present, then that failing is on you, not them. If you can't judge someone's knowledge by asking questions, then you don't know how to come up with the right questions. Again that's on you. The only thing a 4-hr take home test will filter for is desperation. You'll get the most desperate candidates, and you'll do so by wasting days and days of people's time once you add it all up. It's just utterly disrespectful to demand these silly homework problems. I always take it that way. I take it personally, tbh. I find it absolutely insulting to even be asked and I simply refuse.
I'm surprised this seemed to be voted down. I've been a hiring manager for over 30 years, and I never do "technical tests" -- no take-home, no live coding, none of it.
I have a map of topics and questions, and I get the candidate chatting about their past projects, their approach, and what they liked/disliked about past projects and various technology they've used.
It takes a maximum of one hour and usually close to about thirty minutes to make a yes/no decision on a candidate (sometimes it only takes ten minutes to make a no decision, and then it's a matter of trying to politely end the interview).
I've interviewed hundreds of candidates this way over the years, and everyone I've hired has been capable of doing the job. Not once have I ever had to let someone go for lack of technical ability.
Part of the problem is that we don't train people how to conduct interviews, and another part is "this is how I was interviewed, so this is how I'm going to interview other candidates" -- pure inertia.
As an industry, we really need to do better.
As for the OP, _if_ I had been administering a vague take-home project that had a 1-week delivery deadline, and a candidate peppered me with Qs and then presented a full proposal for the project for approval, prior to working on it... I would have rejected them. But I'm pretty certain I would have decided to reject them in my regular 30-60 "chat" interview, and I would not have moved on to the take-home project and wasted their time like that. So, again, I fault the interviewer(s) for not being able to filter candidates efficiently.
I had an interview once to work for IPFS, with a guy who already knew my skill level well (decades of experience), because I was active on their web forum solving everyone's problems, for about a year. Even during the interview he mentioned the technical part of the interview wasn't even necessary, because he had total confidence in me, and didn't even ask technical questions at all. He spent the ENTIRE hour selling me on the job. I hardly had time to speak. Everything went perfectly fine. Then he asked me to do the take-home project at the very end, just because "everybody has to", and I politely told him sorry I don't do those. So that ended the opportunity. So you're exactly right it's absolutely an "inertia" thing.
The solution to this problem ultimately needs to be a regulatory one.
People should be paid for the time they spend in interviews.
You make that the law and this ambiguous hoop jumping bullshit goes away real fast. Companies will optimize for cost instead of dumping cost onto the prospective employees.
This is good for the economy because it forces companies to innovate and optimize the interview process and it saves hundreds hours of totally economically unproductive time on the part of candidates.
That's basically my feeling too. No one would ask for a maid to clean their house free for 4 hours, in order to decide if they want to hire her long term. And that's precisely what hiring managers have been getting away with for decades, because no one is pushing back. The managers are essentially abusing their power. It's all about abuse of power and utter arrogance.
I don't know what ETL means (I could look it up) - regardless I can do that ETL challenge. Realtek SDR dongles and Python loved messing up CSV. The only question I'd have is if a null or and "incorrectly formatted" row was invalid or not.
So I have to wonder, is this a junior position, or did too much hadoop rub off on me a decade and a half ago?
This is for a medior solution, but it’s more intended as a data engineer’s “fizzbuzz” and then in the follow-up call I’ll ask them how they would distribute the workload over multiple clusters, make it idempotent, etc — that’s where the real technical evaluation happens.
How do you know that interviewees aren't spending more time on it?
Because you can't guarantee all candidates are spending the same amount of time, it becomes a game theory problem where the candidates will typically lose in some form. In many cases, the right answer is to spend extra time making a really polished (but not too polished!) solution and pretend like you stayed in the time limit. And every candidate is either a) doing that, or at least b) worried that their competition is doing that.
Even if we ignore that dynamic, 3 hours is a long ass time for a candidate to spend when they're not even sure they'll get to talk to another human about it.
In a 1-hour interview, you can run a candidate through a programming exercise and be guaranteed they're not wasting extra time on it. And if they happen to prefer doing take home assessments, you can always let them send you an updated answer later. (But often by the time a candidate asks me if they can do that, I've already developed a favorable view of their skills and can tell them, "go for it if you want, but you've already 'passed' my test.")
By keeping the candidate-interviewer time investment the same, you guarantee that you're respecting the candidate's time as you would your own (because you're sitting there with them.) I can help them skip over the parts I'm not interested in (e.g., by feeding them info they'd be able to find via search or telling them not to worry about certain details.)
If a hiring manager doesn't respect their candidates' time, how likely are they to respect their employees' time?
Yep, this is what I am taking from this thread: Next time I am given a take-home, I am going to ask them to promise, that I will get to talk to a human about it. They can of course straigh-ass lie about it, and I am sure I will run into such abysmal behavior at some point.
> How do you know that interviewees aren't spending more time on it?
In some cases we roughly timed it, scheduling an email for a time the candidate wanted and asked them to return it 3 hours later. In some cases we just treated it as an honour system. We made it clear that the task was intended to take about that much time and that spending more time was not allowed/encouraged.
In reality, we found that good candidates took ~1-2h, and in some cases where candidates spent a lot longer and owned up to it, we found no improvements. In one case a candidate submitted at 3h and then again at 8, and we marked the 8h version 1 mark lower.
I agree with the author that live code reviews are much better than live coding, but if companies insist on a live coding exercise: let me bring my laptop and mouse leave the room for 45 minutes and come back when we can talk about what I built.
If this had been limited to 3 hours then the worst case is that the candidate would have lost 3 hours, but far more likely is that they would have come up with an entirely different proposal and/or solution that was appropriate for that timeline, and that extra information would have made it clearer what the company was looking for.
The other point I'd always encourage applicants to confirm is: are you looking for any answer, or are you looking for a good answer. Some take-home tests are purely about passing a test suite and how you manage it doesn't matter, some would prefer that you meet 80% of the requirements but write better code. I've seen applicants do the wrong thing on both sides of this.