I have run more than one thousand interviews in my career. Does that make me an expert? Not necessarily. But it does mean that thanks to good mentors (and some not-so-good ones), multiple organizations, and tons of candidates, I have seen many, many different techniques. One of the best, is the audition.
What a Programming Audition Is Not
Folks have argued about what a good or bad software developer interview is a ton. Many readers may already have a picture formed of what an audition is. Since that may differ from what is described here, let’s start by setting out what an audition is not:
- whiteboard programming
- “Aha!” problem-solving
- required to be solved to be hired
- to be done alone or “take-home”
- unpaid work
Let’s go through those one-by-one.
Whiteboard Programming
Most of us have been on one side or another of a whiteboard programming exercise. These can be useful. But they are often done in ways that cause undue stress for the candidate. Additionally, when not run well, candidates that do well are actually less desirable as hires than those that struggle or fail. In fact, this is such an easy technique to get wrong that it is recommended that it not be used at all.
The stress of whiteboard programming often comes from the requirements placed on the candidate. For example, requiring 100% valid syntax or an interviewer who acts like a compiler more than a person. Very few people need to be able to write syntax-perfect code without the help of some kind of code-aware editor. And even then, syntax errors are among the easiest to find and correct. Failing someone for these type of unrealistic—as in not in line with the reality of the job—expectations eliminates candidates that could have been successful as employees.
What is being tested here is whether the candidate can remember syntax and solve contrived problems. And to be solvable in an interview on a whiteboard, all of these problems must be carefully contrived. This activity is so far removed from being able to understand the actual technical and business challenges an employee faces that there is essentially no correlation whatsoever. There are books of these contrived problems that someone could study. If they happen to have memorized the solution to the ones asked, but otherwise cannot code at all, they still pass!
Do I expect that to mean that whiteboard programming will suddenly never happen? No. It is too easy to set up and far too common. Regardless, that is not what an audition should be.
“Aha!” Problem-solving
Those same books that have whiteboard programming problems also often have “logic bomb” problems as well. A popular example is, “How many ping pong balls can fit into a school bus?” The concept is that having the candidate talk through the steps they take to come up with an answer will provide insight into how that person thinks. This is only valuable if the interviewer is very capable of removing their own biases on how someone “should” think. Is a mathematical solution better than an empirical one? Maybe in some cases, but not in all. Trusting the interviewer to evaluate this is a good way to end up with a horrible mono-culture where everyone thinks the same way about things and therefore cannot solve problems that do not fit that way of thinking.
Just as bad are the problems like the “wolf, goat, and cabbage” problem. The claim is that they actually resemble the types of problems one might have to solve when a new algorithm is needed to improve performance. In reality there are several issues. First, the solutions to many of these are readily available. Second, the wording and context of the problems rarely have anything to do with the job and often serve to throw folks off. Third, the “aha!” I am using to name this type of problem indicates how they are usually solved: the solver just happens to stumble across the right direction. Fourth, there are sometimes multiple valid solutions, so what if the candidate provides one the interview did not realize was valid? Fifth, a vanishingly small number of software developers have ever or will ever have to invent an algorithm like this in their career.
Required to be Solved
As can be seen from the previous sections, there are a ton of reasons why a candidate might not solve a problem or might come up with a solution that is not recognized as such by the interviewer. To avoid this being a problem, it is imperative that candidates be “allowed” to fail to solve the problem and still be hired. This takes some pressure off of the candidate and also requires the interviewer to focus on what is supposed to be important: how the candidate actually works.
I have yet to meet a developer who did not run into a problem that they could not solve for an embarrassingly long time only to later realize the solution is amazingly obvious. Sometimes it involves the person stepping away and coming back fresh. Often it involves talking to someone else about the problem. And the worst is when someone walks by the desk and points out a “=” where a “==” should be. These things happen to the best of us. Why punish candidates when that could easily be the case for them in the short time given to solve the problem in the interview?
Similarly, a strict time limit to get to a solution is also a bad idea. Talk about pressure.
Alone or Take-Home
If the org’s developers generally work alone, then shouldn’t we want to see how the candidate does when left alone? As an introvert myself, this is certainly something I might prefer as a candidate.
But. If we head this direction, we make it more likely that the candidate might solve the wrong problem or go off the rails in some unexpected way. We also lose the ability to observe what techniques they actually use, which can be useful so long as lack of a technique itself is not considered negative.
Unpaid Work
Given all of the above, it might seem that having the candidate pick up a story from the Scrum backlog and work on it might be the way to go. This is a terrible idea in most cases. It is exceedingly rare that a story that has so little context required that it could be solved in a typical interview duration with no other onboarding or introduction. If the problem does require several hours, then it is taking more time from the interviewer as well as the candidate. But at least the interviewer is getting paid. So, if that route is taken, at least pay the candidate a reasonable rate for their time.
In my experience, attempting this has generally not been worth the combined costs of setup required, the interviewer’s time, plus paying the candidate. Especially when there is another alternative.
What a Programming Audition Is
Speaking of alternatives, it is time for some payoff, right? Let’s see what actually does make sense for a technical audition. Here are some characteristics to try to achieve:
- match daily work
- be solvable in under an hour
- require technical and/or domain research
Match Daily Work
The point of an audition is to provide information on how the candidate will perform in the position. This means it needs to match both the work the candidate would do in the position as well as how that work would be done. Does the team work with Ruby/Rails? The problem should involve Rails in some way. Does the team generally use RubyMine or some other toolset? The candidate should have access to it. Do employees use web searches to find answers to problems they encounter? The candidate should have internet access.
Note that it isn’t enough to simply provide access to these things. The candidate must be made fully aware these things are available and that there is an expectation—not requirement—they will be used. Most candidates have not done this kind of audition and will assume they are not allowed to use Stack Overflow, for example.
An additional plus is that the candidate gets to see what really would be expected of them in the position. This allows them to make a better choice if and when an offer is made. Remember, a hire is only a good hire if both the org and the employee are happy with it.
Be Solvable In An Hour
Remember, we do not want to take tons of both candidate and interviewer time. This is where things can get tricky in terms of finding the right problem. First, however, it is important that as much setup as possible is done before the candidate starts. For example, if the candidate needs to be able to run unit tests, make sure the unit test suite runs and that the command to run it is very clear and obvious. None of the time should wasted having the candidate set up things that a typical employee would set up once and then never or rarely have to touch.
Require Research
The last thing we want, generally, is to have a problem that someone can come in and finish in fifteen minutes without our getting any real insight into how they work. Avoiding this means making sure the problem has some aspect that the vast majority of candidates are unlikely to have seen. Is there a special library that the org’s development team uses? Make the problem revolve around figuring out how to use that library to come up with a solution.
How to Create and Run an Audition
Now that we know what an audition really is and is not, we can see that it is not necessarily simple to set up and run. Here are a few thoughts on how to bring this technique into an organization.
Use What You have
I mentioned above that the problem can be built around using a library the team already uses. This can be extended to coming up with the problem itself. Ask the team to provide a list of issues they have run into, especially those encountered early in their time with the organization. In fact, gathering a few problems during a team working session is a great way to start. Be sure to let the team know that the goal is not to “stump” the candidates but to require them to do a bit of searching and thinking to solve the problem well.
Simplify
From the candidate problems, select the ones that can be simplified the most while still requiring a good level of thinking and research. This should be very similar to the kind of simplification done when trying to provide an example of how to reproduce a bug when asking a forum question or opening an issue with an open source project.
Use Tests to Define the Problem
Capture the simple version of the problem in a unit test suite with at least one passing and one failing test, preferably more. The passing tests are to make sure the tests are set up correctly. The failing tests make it easy for the candidate to definitively know what they need to do and how they are doing at any given time. (Note: This is also how I prefer requirements be given to developers in daily work. Some degree of test or test specification that the system does not currently pass. Perhaps more on that in a future article.)
Make it Reproducible
Once the problem has been captured in a failing test suite, it is time to put it in version control if that has not already been done. Before the candidate starts to work on the problem, create and switch to a branch specific to that candidate. It can be great to be able to actually commit the code the candidate writes—often with some explanation describing how they did the work—for reference later. Additionally, it makes clearing out the work and refreshing things for the next candidate as easy as switching back to the main line.
Practice
Have existing employees take turns playing the role of the candidate and the interviewer. This is the best way to find the ways the problem does and does not work. If few employees can solve the problem in well under an hour, it is very unlikely a candidate will with the full hour.
Let multiple team members observe multiple practice runs. This teaches them that there are multiple ways to go about solving the problem. However, be careful to make it clear that a candidate does not have to use any of the ways they have seen. And that just because a candidate does use a technique someone else did, that does not make them the same as that other person, whether that would be good or bad.
Deemphasize
As implied just there, the interviewers should be trained to expect this to only be a small part of the overall consideration of a candidate. While this is one of the best techniques for finding out how a candidate will perform, it still has flaws. If the interviewer does not realize that, they can become detached from actually paying attention to all aspects and start to treat this as a pass/fail process where getting the tests to pass means a hire and anything else means non-hire. When that happens, the audition is suddenly no better than many of the problematic techniques discussed earlier.
Similarly, let the candidate know about the audition process before it starts. And make sure to let them know that getting to a solution is not the point. Or at least not the only point.
Interviewers Help, Not Judge
Along those same lines, once the candidate is starting the problem, the interviewer is no longer really “just” interviewing. They are there to help. The role should be considered like a slightly more hands off “back seat” when pair programming. Watch for the candidate getting stuck or going down a bad path. When that happens, give some small guidance. Don’t solve the problem for the candidate, but try to keep them from getting overly frustrated.
Wrap-Up
At this point, it is time to start finding those problems and building out auditions. Remember that the audition isn’t a “pop quiz” or “gotcha” type of thing. And it definitely is not meant to be unpaid work. Make sure it is reflective of what the position requires, requires some effort, and is not a marathon. Make it as simple as possible for the interviewers to set up and run. Ideally even make it a little fun for everyone involved. And finally, to paraphrase the cereal commercials, this is a good part of a balanced interview process. It is not the whole thing.