As I slowly gear up towards finding a new job, the dread of having to go through countless ill-designed interview processes is settling in. There are a lot of aspects of tech interviewing that are broken, and the main culprits are usually clearly identified (looking at you whiteboard ...), but one that might be perniciously so is take-home assignments.
The case for take-home assignments
Take-home assignments are a staple to tech interview processes, where candidates are given some instructions (usually, some functionalities they need to implement in code) and some time to complete said assignment. If this sounds like free labor to you, then it probably is, but more on that later! Take-home assignments have been somewhat popular in tech companies as a way to break from the status quo of whiteboard interviews and automated code tests.
There are some qualities that make take-home assignments one of the less dehumanizing ways of assessing for technical skills. They allow candidates to showcase their skills using their own tools and their own time. They provide more accurate signals by putting candidates in conditions that are closer to what they will encounter in their day to day, and they tend to focus on real world skills, not the capacity of candidates to memorize algorithms and data structures. When designed properly, take-home assignments can provide a positive experience for candidates.
There have been a few studies on interview processes, and work sample tests tend to have a high predictive power, meaning how well candidates that pass the test end up performing, which is good. Take-home assignments in particular, allow to simulate a work environment as close as possible, and provide room to not only assess technical and problem-solving skills, but also other indispensable "soft" skills such as communication and collaboration, which might be difficult to evaluate in a time-bound setting or using medium such as whiteboards.
As good as take-home assignments are, in particular compared with the alternatives, they unfortunately still have glaring shortcomings.
The shortcomings of take-home assignments
We won't go over the typical issues, such as some candidates spending a lot more time than others, hence making it more challenging to compare submission, or even straight up fraud in the completion of the tasks. Instead, let's focus on more insidious limitations of take-home assignments.
They represent a high upfront cost for candidates, who might end up spending 3-5 hours (in best case scenarios) working for free to complete a rather arbitrary exercise. Some examples I had to do in the past include implementing a weather app, a roman number converter, or a favorite TV show web app. In some cases, the task was more closely related to the company I was applying for, but it still required a considerable investment of time from my side. Of course, none of the companies I interviewed with (expect for one) offered to compensate me for the assignment. This high upfront cost of take-home exercises tend to dissuade a significant part of the candidates to continue with the process.
In theory, take-home assignments allow for a thorough and comparable evaluation of the candidate's skills related to the role they are applying for. An important part of the predictive power comes from the design of those assignments. When done properly, they can provide strong signals to companies, and still be somewhat palatable for candidates. Unfortunately, in my experience, besides some assignments having little to do with the actual role, reviews of take-home tests tend to be sloppy, thus failing their very own purpose at gathering high-quality signals. One such review prompted me to write my very first blog post, after being told that the hiring team "didn't like my coding style" regarding my submission to a take-home assignment. And the lack of feedback, let alone actionable feedback, seems to be the norm rather than the exception. In those cases, not only are take-home assignments a poor experience for candidates, they also yield poor results in terms of predictive power. The evaluation of the assignment can also flair up unconscious biases if the people reviewing them don't receive proper training.
Talking of biases, one of the main points against take-home assignments, in my opinion, is that they tend to be fairly discriminatory. They tend to favor candidates who have the luxury of time and the willingness to still code after their full-time job. They tend to favor candidates who have the needed resources (decent personal computer, good Internet connection, quiet and comfortable home environment) to perform at their best. Sometimes, the very design of the take-home assignment can discriminate. It is not so much a level playing field as some proponents of take-home assignments want it to be.
Where to go from here?
Alright, this is all nice and stuff, but where are we going from here?
We've already seen that some alternatives to assess technical skills provide a worse candidate experience and predict less accurately for the job. Whiteboarding and automated code tests are out of the game. Time-boxed technical challenges can also distort the actual skill set of candidates as coding under pressure doesn't correlate much with the actual performance on the job. Other popular methods include reviewing candidates open source code, but this comes with its own set of limitations, as most people don't have time or don't enjoy writing code (basically doing more work) during their free time. Not everyone has time to work on a side projects they can then potentially show during interviews.
So, should we just not assess candidates? Of course, the answer isn't so simple, but let's step back and first look at what we are really trying to assess. As stated previously, at this stage of an interview process we are looking at the candidate's qualifications to do the job, in this case, be a software engineer.
That role roughly involves:
- understanding requirements,
- writing high-quality code (correct, maintainable, performant),
- solving problems (if you happen to join an empowered product team!),
- collaborating at the very least with fellow software engineers (or in some cases, other stakeholders such as product managers, designers, data analysts),
- communicating with stakeholders (deadlines, production issues, ...)
... I'm probably forget a few important roles, but let's stick with those.
If you look at that list of skills and have hired using take-home assignments, you'd say that, in theory at least, they tick all the boxes. We are able to gather high-quality signals on all those skills with a well-crafted take-home exercise! And for the most part, I would agree with you. They are a decent tool, albeit imperfect.
Studies have shown that the highest predictor and less biased type of interview is structured interviews. This means that every interviewee answers the same set of questions in the same order, and that the questionnaires are specifically designed to gather all important signals to make a hiring decision. It's a practice almost as old as the art of interviewing, yet we put that aside for practical reasons. It is far easier to ask a candidate to work on an assignment over the weekend than it is to craft a set of high-quality questions and train your team to conduct high-quality interviews. Yet, this might be our way out of the insanity of tech interviews.
The best predictor of future behavior is still past behavior. Asking insightful questions over multiple interviews, and gathering all relevant signals might yield higher candidate satisfaction and higher predictive power.
Ultimately, each company and organization has to decide what works best for them, but they also need to understand that their choices in regards to how they structure their interview process can have (un)intended consequences in the markup of their teams, the culture of their company, and the overall success of their business.
