What my ideal interview process looks like, revisited

Hiring and building high performing teams has always been an interest of mine. I have always been involved in the hiring processes of all the companies I have worked for, and in some cases even helped revamp those processes. I also have a fairly extensive experience interviewing as a candidate.

As such, I have refined over the years my views on how an ideal interview process for software engineers could look like. This is my latest version after going through a 6-week job search during which I experienced wildly different processes.

How to best interview candidates

Before we delve too deep into what an ideal interview process looks like, let's see why it is important to craft a thoughtful interview process.

The main role of an interview process is to assess candidates' skills to hire the people that will perform the best at the job. It is quite telling then that 40% of new hires leave their jobs within the first 6 months, and more than 50% do so within their first year. Even though those numbers look at the general population, we can draw a parallel to software engineering as well.

That means that most interviews seem to be fairly poor predictors of job performance. That might partially be due to the ubiquity of unstructured interviews. When no thoughtful interview process is in place, unstructured interviews tend to be the main tools at the interviewer's disposal. They are easy to setup and they allow interviewers to move the conversation in any direction they see fit. They feel more conversational and less scripted, and provide flexibility in which signals to gather.

Nevertheless, studies have shown that they are terrible predictors of performance (only predicting for 14%-31% of a candidate's success). They are fraught with biases, are inconsistent between candidates, and are almost purely based on gut feeling. Interviewers are confident about their conclusions after unstructured interviews, yet studies (Highhouse (2008)) have shown that they are the least predictive tools in interviewing. And, not only are unstructured interviews almost irrelevant, but another study (Dana, Dawes, & Peterson (2013)) has shown that they can be harmful to the decision process.

Consulting firms, then tech companies, popularized brain-teaser questions such as the infamous “How many ping-pong balls will fit inside a 747?”. Those types of questions can not only alienate candidates (Ali et al. (2016)), but they have been found to have no predictive power by internal data analysis at Google.

Another trend that tends to backfire as well, is copying what top tech companies are doing. In particular, picking up ideas from processes where candidates go over an excessive number of interview rounds. Those lengthy interview processes not only impoverish the candidate's experience, but research at Google shows that beyond 4 interview rounds, there is substantially less predictive value added by the subsequent rounds.

As far back as the late 1990s, there has been clear evidence that the best predictor of job performance is cognitive ability (Vinchur & Shlippmann (1998)) and that general mental ability can explain up to 70% of the variation in job performance.

The canonical study on top of which Google based its hiring practices, Schmidt and Hunter (1998), shows that the best predictors of job performance are, in order of validity:

• Work sample tests
• General mental ability tests
• Structured interviews
• Peer ratings
• Job knowledge tests

Subsequent studies (Hunter, Schmidt, & Le (2006) and Schmidt, Oh, & Shaffer (2016)) now show that general mental ability tests have higher validity than work sample tests.

With that information in mind, let's see how we can craft an interview process for software engineers that yields high predictive power while providing a great candidate experience.

Disclaimer: this is my personal opinion where I'm trying to balance my personal thoughts with the science of how to best interview, the candidate's experience in going through a hiring process, and real-world constraints on a company.

Setting your hiring process up for success

Crafting an ideal interview process starts well before the first interview!

As a hiring manager, you need to ensure that you can communicate clarity on the role(s) you are hiring for. Crafting an inclusive job description is also essential to attract diverse candidates. Finally, it is important as a company to provide training to interviewers, as interviewing is not a skill that most people have learned in college or practiced regularly.

Define the signals you are looking for in candidates

Beyond the organizational imperatives needed to budget and open up a new position, it is primordial to understand what profiles are needed to fill in said position. As such, the first step should be to define which are the requirements of the position, and by extension, which are the important signals to look for during the interview process.

A good technique to uncover the requirements of a position is to use the MSN list, which helps you focus on what really matters for the role. The MSN list is composed of 3 elements: Must, Should, and Nice. Under each section, you can add up to 3 attributes. Those attributes need to be atomic, specific to the role, and you should be able to evaluate them during the interview process. They will serve as the basis for the job description and the candidate's assessment. You can check an example of how to write an MSN list in this tweet.

Other key signals are the company's values, and the skills present in your tech career ladder. Clearly stating the company values as elements on which the interview will be collecting signals allows for a more accurate assessment of "culture fit", which doesn't rely on individual preferences and biases. Using the skills defined in your tech career ladder, which are probably also used in performance reviews and promotions, helps keep interviews centered around the actual skills needed to perform in the job. It also provides clear guidelines on leveling candidates.

Defining the signals you are looking for in candidates is a useful exercise in crafting an interview process that will yield the best results. It serves as a basis for the job description, but also clarifies which signals are important, and it drives the design of the interview process and the assessments of candidates.

Write an inclusive job description

Diversity is a topic where tech in particular is still not at the level it should be. Even if only for that, diversity makes business sense, with diverse teams outperforming non-diverse teams by up to 35%. For more than 75% of candidates, diversity is an important factor when evaluating companies, and nearly a third  of candidates wouldn't apply for a job at a company which lacks diversity (me included!).

Sure, there are many challenges in recruiting and retaining a diverse team in tech, but there are also some straightforward solutions. To stay close to the topic of this article, we will only focus on inbound candidates and how to ensure we don't alienate diverse candidates in the job description and the interview process.

I have only once in my career been in charge of writing a job description for a software engineering position, so as my mileage here is probably quite limited, I'll make use of some articles from people more in the know. Some recurrent points to keep in mind when writing an inclusive job description include:

• Watch out for gender-coded words, such as "hacker", "rockstar", or "ninja" and instead use descriptive terms such as "software engineer".
• Use gender-neutral language such as "you" or "they" instead of "he" or "she".
• Limit the list of requirements to the few "must-have" elements you have identified in your MSN list. Women tend to apply for jobs where they meet 100% of the listed requirements, whereas men tend to apply when they meet about 60%.
• Put some effort in structuring your job description for clarity and exhaustiveness. Candidates spend less than a minute scanning a job description before dismissing it.
• Emphasize your commitment to diversity, equity, and inclusion.

Some other elements that I believe should be present in the job description include:

• a short description and a bit of context around what the company does,
• an exhaustive description of the tech stack (but not as a requirement),
• a clear description of the hiring process and how the candidate will be evaluated,
• a list of your company values (or a link to the list!).

I've also seen great job descriptions where the company went further and described some of the latest tasks software engineers completed.

Finally, go over a few rounds of feedback within your team before posting your well-crafted inclusive job description.

As a quick aside, you want to make it easy for candidates to apply. So beyond just the job description, make sure that whichever tool you are using to get inbound applications is streamlined and requires the minimum amount of information to apply. Unless you are suffering from large amounts of applications, don't put artificial barriers such as asking potential candidates to answer a handful of questions with their application.

Provide training to interviewers

Training is essential not only to ensure that interviews are run within the limits of legality, but also to ensure that candidates have a good experience, and more importantly that the interviews yield high-quality signals that allow them to make pertinent hiring decisions free from biases.

Unless you have been a recruiter in your past life, chances are you have never had any formal training in how to interview candidates. If you are lucky, you might have had some legal training around what you can say or not and what you can ask or not to candidates during an interview.

Unfortunately, looking at my own experience and others, it seems like very few companies provide interview training. As I don't have any experience in interview training (other than shadowing other interviewers), I'll just link to this guide.

Interview process

After setting the groundwork for what an ideal interview process looks like, let's take a closer look at the interview process itself!

The main goal of interviewing is to assess candidates' skills against the requirements of a job. At least in software engineering, that needs to be counterbalanced with a great candidate experience and a certain speed of execution, as talent is scarce and the competition to hire them is ruthless.

Here is a summarized version of what my ideal interview process looks like:

Note that the technical, the system design and the leadership / behavioral interviews can be scheduled the same day (which would total 3 hours of back-to-back interviews). That helps accelerate the process, and it provides all signals to make a decision instead of rejecting candidates after a single round where they might have had a poor performance.

A quick word on reference checks: I personally don't know how much value they bring to the process as they are mostly going to suffer from confirmation bias. Instead, I believe that reverse reference checks are a more useful tool, and the interview process should have already uncovered any potential red flag that a reference call might uncover.

Overall, the above hiring process is composed of 4 rounds of interviews. It involves up to 10 people from the company. A successful candidate will spend 5.5 hours in calls during the process, and the process from application to offer can be completed within 2 weeks.

Now let's look a bit closer at each of the stages in the hiring process:

CV Screening

This stage is the one where the pool of candidates will be whittled down the most. Based on recruiting data, the conversion rate between application to interview sits at 12%. My own experience screening through resumes does seem to validate that. As such, this stage is one of the most crucial in terms of qualifying good candidates. At this stage, you probably want to minimize false negatives, meaning that you want to minimize the number of potentially good candidates that were rejected.

Ideally, thanks to the clarity brought in by the MSN list, a recruiter should be able to filter candidates with high accuracy against the job requirements. It might in some cases be useful to loop in the hiring manager for a second filtering of the candidates, but regardless of the setup, make sure to qualify candidates within 2 days of their application.

As much as you would like to automate this phase with some clever AI tool, please don't! Not only is rejecting candidates with an automated tool possibly illegal in some jurisdictions, but screening resumes is a hard problem to solve and using automated screening tools might lead to more false positives (people "cheating the system") and false negatives. Also, the reality is that most tech companies don't seem to use such tools, and one of the main studies on which this belief is based turned out not to be so sound.

Something that you should definitely automate is an email response to candidates that you reject at this stage. It is surprising to see that a majority of companies still prefer to ghost candidates, when a simple automation can go a long way.

First Call

Time to schedule the first call with qualified candidates!

Depending on the volume of candidates and the current size of the company, this might be done by a recruiter or by the hiring manager. I personally preferred the first calls where I got to chat with the hiring manager right off the bat, as it allowed me as a candidate to get a clearer picture of the company, the team, and the role.

The call should be about 30 minutes long, and it should be scheduled within 1 week of the candidate's application. The goal of the call is to provide more context about the company, the role, and the hiring process on one hand; and assert that the opportunity matches what the candidate is looking for on the other. There is no point in wasting time trying to assess the technical skills of the candidate at this stage, as it has already been done at the resume screening stage, and it will subsequently be assessed further down the line. As you are going to be asking the candidate for 5 more hours of their time throughout the hiring process, make sure to sell your company and the team.

Luckily, your company has implemented transparent salaries and/or all the job descriptions that you post include a salary range for the position candidates are applying for. As such, there is no need to ask about their salary expectations during the first call, as it is fair to assume that, if they applied for the role, their salary expectations are within that range.

As in every other round, use structured questions to evaluate the candidate and provide some room for the candidate to ask questions at the end. Unless there is an obvious mismatch on either side, this round shouldn't slim down the pool of candidates too much.

Technical Round

A lot has been said about how to best assess the technical abilities of software engineering candidates, and people tend to hold strong beliefs that their preferred way of interviewing candidates is the best way to do so.

The truth is that, like most things in life, it depends. It depends on which skills you truly want to assess, and on which signals you believe are important to gather to be able to make an informed hiring decision. As such, I won't go too much over how I would interview software engineers (maybe a topic for another blog post?), and instead I'll try to outline some first principles on top of which to build your own technical round and which are the trade-offs of selecting different methods.

One method that has been (and still is) quite popular is whiteboard technical interviews. Nowadays, it would mostly happen in a shared IDE instead of an actual whiteboard, but the idea stays the same: ask candidates a technical question where they need to write some code in a given amount of time and observe their performance. It might sound like a good way to gather signals on technical and problem-solving skills, which are usually core to the role of a software engineer, yet a recent study found that live-coding interviews mostly filter for candidates who are able to handle or mitigate performance anxiety. That is probably a good skill to have for actors or public speakers, but it's arguable if that should be the primary discriminating factor for software engineering positions. That study also showed that live-coding interviews might be riddled with biases, negatively impacting diversity. Another consideration is that some of the typical questions being asked in those interviews were created not to look for problem-solving skills, but for very specific tech skills that might no longer be relevant today.

Alternatives to those interviews include take-home assignments, talking through past work (for example: some open-source code on GitHub, or a past project), or pair programming sessions. Each of those alternatives also have their pros and cons. Take-home assignments remove the performance anxiety of traditional live-coding interviews, but might still have some inherent biases. Yet, when done right, they can be a good way of assessing technical and soft skills. Having candidates talk about some past work can be a good choice, as past performance is a good indicator of future performance. The issues with that approach is again that it might bias against people who don't have time (or the willingness) to write code outside of work, and it can be more difficult to have a level-playing field in terms of evaluating candidates. Finally, pair programming sessions are an interesting compromise on the traditional live-coding interviews. They are still time-limited and require candidates to write code in an artificial environment, but they can quell performance anxiety when done right, and assess candidates more holistically. In my experience though, some companies just rebrand their live-coding interviews as "pair programming", yet still interview candidates the same traditional live-coding way.

The approach I would take for technical interviews is as follows:

• Clearly define which signals are important and how to best assess them.
• Provide flexibility to the format of the interview, allowing candidates to choose between a pair programming exercise, a take-home assignment or a past project walk-through. In all cases, provide clear guidelines on what you are going to assess the candidates on.
  • For a pair programming exercise, I would schedule a 1-hour call (45 minutes for the exercise), but with the questions being sent to the candidate in advance. Also, make the exercise an actual pair programming exercise where the interviewer works with the candidate to arrive at an optimal solution.
  • For take-home assignments, I would make the exercise small enough that it can be completed on average in less than 2 hours. I would then schedule a 30-minute call to discuss their solution.
  • For existing code, I would schedule a 45-minute call to discuss their code and their thought-process. The risk here is that the candidate presents a project which doesn't allow for high-quality signals to be gathered.
• Make the exercise as close as possible to what the actual work will look like. In particular, software engineers tend to read code more often than they write code, so it would make sense to design an exercise where candidates need to understand and work within an existing code base.
• Always have exactly 2 interviewers, one leading the interview and the other one shadowing.

Providing candidates multiple options for the technical round might not be a scalable solution if you are hiring hundreds of engineers per year, as it requires more set-up work and training on your side, yet it can provide unique candidate experiences, where every candidate can showcase their skills to the best of their capacity.

System Design

I have less experience giving and taking system design interviews, but they always seemed a bit fuzzy to me, which is probably not ideal in terms of unbiased assessment. These kinds of interviews are a staple in big tech hiring processes, and are usually given to more senior candidates (which is maybe why I only recently started seeing more of those in my interviews!). They are open-ended questions about architecting a system and they are supposed to assess problem-solving skills, critical thinking, and experience in the field.

I couldn't find any studies (or think-pieces) regarding the predictive power of system design interviews, but my intuition is that, due to their open-ended and conversational nature, they would fall prey to the same issues that unstructured interviews have. That is even more the case if the questions asked are as vague as in this video (TL;DR: optimize a system with a client, an API, a server, and a database), and where the expectations of interviewers for a successful performance by the candidate might be that they "learn something new".

Instead of a traditional system design interview, I'd propose to better define what this round brings to the whole process. Ideally, every round should add different signals overall so that time spent interviewing is optimized. As such, it might not make sense to look for problem-solving skills again if that has been taken care of in the technical round. Instead, some skills that might be important for software engineers of any seniority, and that can be assessed at this stage, are understanding a given architecture and its trade-offs, written communication, and product sense.

The way I would conduct such an interview is as follows:

• Clearly define which signals are important and how to best assess them.
• Provide the candidate an existing system, ideally related to the products your company is building, and give them a prompt to either improve or add some features to the system. Please make sure that such improvement or feature has already been implemented. Interviewing shouldn't be free labor.
• The expected deliverable is a written document explaining which changes are needed for the system to support the new feature and their impact on user experience.
• Allow candidates to either do the interview asynchronously (take-home) or live.
  • In the case of a take-home assignment, schedule a 30-minute call for them to present their work and to discuss their deliverable.
  • For a live session, schedule 1 hour, but with the questions being sent to the candidate in advance. During that session, candidates are expected to walk through the changes in the system and generate a written document explaining those changes.
• Always have exactly 2 interviewers, one leading the interview and the other one shadowing.

By using a structured approach to system design, it is easier to fairly evaluate candidates across the board. I will dive deeper into what such an exercise could look like in a subsequent blog post, but hopefully you get the gist of what this round should be achieving in regards to assessing candidates.

Leadership / Behavioral

Similarly to system design interviews, I have very little experience conducting such interviews, so I can mostly only speak from my experience as a candidate. As such, this guide can be helpful in covering some aspects of behavioral interviews.

In particular, the goal of behavioral interviews is to uncover past behaviors of a candidate. To generate high-quality signals that are comparable, it is best to use a structured interview process, asking all candidates the same questions in the same order. Also, don't ask hypothetical questions, and instead ask questions based on the candidate's past experience. I'd argue as well that all candidates, regardless of seniority, should go through a behavioral / leadership interview. In this interview, the main signals to look for are the adequacy of the candidate's responses with the company culture and the "soft skills" that have been identified as must-have for the position they are applying for. The basic premise of behavioral interviews is that past behavior is the best predictor of future behavior. Going one step further, it can help determine desirable traits such as self-awareness and a growth mindset.

There are many examples of what questions to ask in behavioral and leadership interviews available online, so I won't be looking too much into the content itself. In terms of format, I would advise for a 1-hour call with, again, 2 interviewers (one leading the interview, and the other shadowing). Ideally, the lead interviewer should be the hiring manager, and the shadow should be from roles adjacent to the one the candidate is interviewing for. For example, when interviewing a candidate for a software engineering position, it might make sense to have a PM or a designer shadow the behavioral / leadership interview.

Internal Debriefing

In most companies, the hiring manager looks at the notes taken from the interviewers during the hiring process and makes a hire decision. That might scale well, and work well if note taking and score cards are clearly defined, but it might be sub-optimal in holistically assessing candidates.

Instead, something that I believe more companies should do, is organize an internal debrief session with all the interviewers (and shadowers) that took part in the process. During the session, each interviewer and shadower (including the recruiter or main point of contact) shares about their interview and the signals they have gathered. They should also provide some guidance on either hiring or not the candidate, and at which level. If there are any mismatches on the guidance (which might often be the case), then the floor is then open for discussion and the hiring manager then synthesizes the discussions in a hiring memo. The ultimate hiring decision still lies on the hiring manager, but this exercise allows for everyone involved to have a say, and provides the hiring manager with a more holistic view of the candidate's performance.

Ideally, the debrief should be scheduled the same day as the last round of interviews (or the day after), and shouldn't last more than 30 minutes. If a hiring decision cannot be made by the hiring manager, the discussion can continue asynchronously on the memo itself, but usually, if it's a "maybe" then it's probably a "no".

Feedback Call

After a hiring decision has been made, it is time to schedule a feedback call with the candidate.

It is important to do this regardless of the hiring decision, as the candidate has put a certain amount of time and energy into the process. In my experience, too few companies provide any feedback at all, and only one company took the time to jump on a call to give me feedback and explain why they were not going to make me an offer. Giving critical feedback is not easy, but it is an important skill to master.

Thanks to the well-structured hiring process the candidate has gone through, it should be relatively straightforward to provide them with actionable feedback regarding their performance at every stage of the interview. Ideally, the call should be done by the recruiter or the hiring manager, and shouldn't last more than 30 minutes. The very first information that needs to be communicated during the call is the hiring decision. Then, ask the candidate if they want to receive some feedback.

From a company's perspective, there are also some benefits to having a feedback call as part of the hiring process. For one, it forces the hiring decision to be based on tangible and measurable traits rather than "gut feeling". It also strengthens the company's brand, regardless of if the candidate will end up joining or not, as it shows that there is a strong feedback culture in the company, and that is usually a good basis for transparency and growth.

Reverse Reference Calls

I have written an article about reference calls and their shortcomings, and why I believe more companies should be offering reverse reference calls (calls between the candidate and existing employees for the candidate to ask them questions). The existence of this round in my ideal interview process is therefore in line with those thoughts.

The idea of reverse reference calls is to allow the candidate to ask more questions about the company and its culture. At this stage, the hiring decision has been made, so whatever is discussed shouldn't be used in changing the decision (unless major red flags show up, but then you probably would like to review your process to understand why they were not picked up earlier in the process).

There are many ways of organizing these calls, but one that I found particularly interesting was from a company I interviewed with. They were proactive in setting up a 1-hour meeting where I would have the chance to talk with 3 employees of the company I haven't interacted with previously, each call lasting for 20 minutes. That format made a lot of sense to me.

It is totally fine for candidates to decline participating in this call if they think they have all the signals they need on the company to make their decision, but I believe companies should be more proactive in proposing such calls.

Offer Call

Finally, the last step in the hiring process: the offer call!

I've never made any offer calls myself, but I have been on the receiving end on more than a dozen occasions by now. Some are just a traditional listing of the salary, stocks, and other benefits, but some that I really enjoyed actually involved a presentation led by the recruiter in which the hiring manager also presented in more details the team and projects. In a way, I believe this should be more of a sales call than the beginning of a transactional negotiation.

If your company has transparent salaries per level (which it should have!), then there is little negotiation to do, as you present your offer in line with your own formula. If your company doesn't have transparency in regards to compensation, now can be a good time to look into it, as there are some clear benefits (and some downsides) to salary transparency (if it wasn't clear enough, I am a big proponent of transparency!).

Back to the offer call, it should be a 30-minute meeting involving the recruiter and the hiring manager. Ideally, the recruiter will present the compensation package and explain the benefits, and the hiring manager will sell their team and projects to the candidate. After the call, the offer should be sent in full to the candidate, with the option to have another call in the future if needed. Clear next steps should also be included in the email so that candidates know what will happen if they accept the offer.

Hopefully, the candidate feels as strongly as you about there being a strong fit, and y'all go on to do great things together!

Parting thoughts

If you've read this far, congrats! This ended up being a longer blog post than I first envisioned. Before we wrap this up, here are some overall parting thoughts:

• Be transparent about the hiring process, and provide pointers as to what skills are going to be assessed and how.
• Be respectful of the candidate's time. Arrive for the interviews on time and well-prepared (for example: having read their resume), which shows that you care about the candidate and value their time. A line that I've used as an interviewer at the start of every call is: "thanks for taking the time to interview with me".
• Be aware of your biases and actively try to limit them.
• Don't base your hiring strategy only on inbound candidates, as that might lead to a lack of diversity in the candidate pool, and ultimately your company. Instead, also invest in reaching out to great candidates. Probably worth another article!

Despite the misleading title of this article, no interview process is ideal. What sounds like a great hiring process to me might not be directly applicable to your company. And whatever your current hiring process is, chances are it can be improved as well. The only constant here should be change through iteration. Go back to first principles, try new approaches, measure their impact, and make the required changes to keep improving.