Show Notes
You are a hiring manager and your organization has given you the green light to hire more people. What do you do next? Many people might feel the answer is obvious but for a surprisingly large number of companies, their hiring processes actively impede them from building great teams. Andy and Mon-Chaio pull apart conventional hiring practices to separate the wheat from the chaff and provide tactics to help guide the creation of a truly impactful hiring process.
Opening quote from Job interviews are a nightmare — and only getting worse.
References:
- LeetCode
- Red-black trees
- McKinsey outer loop
- What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring
- STAR
- Normalization of deviance
- Albert Bandura
Transcript
Mon-Chaio: Thank you all for joining us for another episode of the TTL Podcast. A couple episodes ago, Andy and I talked about the concept of hiring, from the lens of do you really need to hire, some new work comes in, we need to get it done. Engineering says we need to hire.
Sometimes leadership says, do you really need to hire? So we talked a lot about that a couple episodes ago. Today we thought we would talk about the mechanics of hiring and the tactics of hiring. So now you know, or you’ve decided as a group that you need to hire some extra people. how do you go about it?
What are some of the best practices? What are some of the things that people do that maybe are not best practices or are anti-patterns that maybe are still really prevalent in the industry? Anything to add, Andy?
Andrew Parker: I think we can just end this episode right now and we just say you do a whiteboard interview. You have them write some assembly code on the whiteboard, and that’ll find you the best programmers in the world. Can we just leave it there?
Mon-Chaio: Yeah, absolutely. So, until next time, stay kind …
Andrew Parker: This was the easiest one ever! But yeah, I think we all know now that the whiteboard interview is probably not the end all, be all of the hiring process. And for the new leaders listening, or maybe the old ones who just want to hear another take on this, there is so much more to hiring than just the interview, just what happens in that room.
So maybe Mon-Chaio we start out by just doing a quick rundown. What else is part of hiring and what are we gonna try to cover today?
Mon-Chaio: Makes sense Andy. There’s definitely a lot of parts of the interview. I think where we’re focusing really today is thinking about if you are a hiring manager, and you’re thinking about setting up an interview process, what are the things that you want to get right immediately? What are the preparations that you have to do to make sure that the interview goes smoothly?
Now, for some of our listeners that may be working at larger corporations, you might think “I don’t really have control over that, my company or my corporation already has a hiring process and I just have to follow it.” Which may be true, often it’s true, but we’re hoping that there’s still some tactics that you can take away from this, even though it’s probably more geared towards a smaller organization or some AAA part team of the organization where you really can control your hiring process.
Andrew Parker: Because in those smaller organizations, you really do have a lot of control over how you run this. You’ll have interaction with your HR team and you’ll have interaction with managers around you. But if you are the engineering manager in one of those smaller organizations, you have a lot of control over your hiring process and how you set up that hiring process early on will have a lot of influence over what that company turns into. Because in going back to our culture episodes, the people that you have around determine a lot about the culture that you’ll have and your hiring is a big aspect in this. So we’re gonna be covering what does a hiring manager do? We’ll talk a bit I think about thinking through what it is you care about for hiring. What are you hiring for and what are you not hiring for? What are you trying to keep out of your criteria? I think we should get a little bit into biases. And then we’ll take a little bit of a foray into who should make the decision and how to think about what’s going on in an interview. So you’re gonna have people coming through, you’re gonna be doing this interview. Let’s talk a little bit about what kind of mindset should you have in talking to candidates.
Anything else you think Mon-Chaio?
Mon-Chaio: I think that’s a lot. But let’s dive right into it. We said we’d start with, well, what does a hiring manager do? But that ties into the first thing that we really care about with hiring, which is, what do you care about and why, and what don’t you care about?
Because you can’t design a process without knowing what it is that you’re looking for. And we foreshadow this a little bit by saying, well, we both don’t really agree with whiteboard interviews, but that’s not really the case in industry. There’s some really famous books out there written that are really big on whiteboard interviews, and without me getting into the specific details, ’cause I don’t want to really rag on people.
During this podcast, I was at a company, a large company that was transforming their hiring process in the recent past. And they brought in an author of one of these books who was a big proponent on, LeetCode type questions on the whiteboard as a central part of hiring engineers. So I think it really does all start with
What is it that you are looking for, and what is it that you’re not Before you can talk about how
Track 1: you’re
Mon-Chaio: gonna assess, so what are you looking for, Andy, when you’re hiring?
Andrew Parker: I’m looking for someone who I want to go down to the pub and drink a beer with.
Mm-hmm.
that’s
not what I’m looking for. If that is how you set up your interview is like, Hey, let’s just sit down and have a chat. You have to think that’s what you’re then designing it to towards. So this gets to that. What you’re asking macho is like in my mind, you ask during an interview as closely as you can, what they’re going to be doing for their job. And this is why I dislike whiteboard interviews because for the most
part, I am not writing code on a whiteboard. Now, I might sketch out ideas on a whiteboard, I might do some design work on a whiteboard. I might do that in a particular way, though I might do it in a collaborative way. might not do it in a collaborative way. That comes down to what kind of a team you want to have. So, for instance, I actually personally could be fine with a whiteboard interview if it was about, collaboratively exploring a domain and a problem space and sketching out a design.
So in that case, yes, a whiteboard could be part of an interview for me, but it’s because I would, be looking for that particular way of, behaving and interacting and operating, what else would I be looking for? Well, of course, just writing code. think
Mon-Chaio: if you’re hiring an engineer,
Andrew Parker: if I’m hiring an engineer, if I’m hiring a tester, looking for, can they find problems in a system, if I’m hiring a QA person, I would probably be introducing them to a thing that we’re working on and working through it with them about how we could improve the quality of it. Like what kinds of steps we could take to improve the quality, quality assurance. And so it’s like, what would I want this person to be doing for their job? Mancho anything that comes to mind for you that you would try to care about or not care about? And you have ideas about how you might assess that? How you might look for those kinds of things?
Mon-Chaio: Let’s touch on that a little bit later actually. We get to the biases, we might talk a lot about like interview technique and so that might be a good chance to talk about that. Something that you’ve heard already early on from the both of us though, is I think we both really believe in tailoring the interview process as closely as possible to what they’re gonna be doing on a regular basis. that a reasonable framing of your opinion, Andy?
Andrew Parker: Yeah. Yeah.
Mon-Chaio: There are people that don’t believe that, and I can absolutely see why. And so what they recommend is this idea of having a proxy to that. So they say, well, it’s not important that the interview process be aligned with what they would do daily or how working at a company is, but we can use artificial interview means in order to proxy.
Those skills and see how relevant or, how well developed those skills are in the individual. For a lot of people that are proponents of whiteboard interviews, that’s what they would say that the whiteboard interview is a proxy towards problem solving.
And the reason why I think that’s reasonable is I don’t really think you can assess all the skills of an individual or assess all of their working skills on a day-to-day basis in however long your interview loop is. And we’ll talk a little bit later maybe about the length that we feel inter loop loops should be.
And so getting back to the, what are you looking for and what are you not, it’s kind of like the culture thing we’ve talked about, right? I think you have to identify the top 2, 3, 4, or five things that you really do care about. Is it really coding skills in a vacuum?
Andrew Parker: Mm.
Mon-Chaio: Is it really being able to sit down for an hour by yourself and write something without the use of Google?
Is it that you want them to be able to solve a tough problem without bouncing ideas off of other people? A lot of whiteboard interviews test for that, right? It’s the recall, oh, do you remember your red BlackTree algorithm from CSS class? And can you pinpoint that this pattern is the right pattern to solve the individual problem while nobody else is helping you out?
That a great proxy for what you want them to be doing on a day-to-day basis? I think it’s important that you pick, know, those core sets of things, and then design that to be as close to everyday working as possible so that you get the best signal possible.
Andrew Parker: Absolutely. And that’s not to say that asking those kinds of questions is wrong. It’s all about that context. Perhaps you are an organization where, for instance, people do work on their own. You’re very remote and you’re async. And for the most part, they’re not going to have, uh, immediate collaboration with others.
And so what you want is you want people who can just sit there and kind of work their way through it. On the other hand, you might be an organization that is very much about constant collaboration that, uh, maybe never has really complicated algorithmic problems. And so asking someone about red, black trees, and just sitting and staring at them for an hour as they work it out isn’t going to be a very good, representation of what you’re looking for. So I think in a lot of cases we become very shallow or we can end up accidentally being shallow about what it is that this job entails. And we say, oh, you’re a software engineer, so that means that you just sit and write code, or, oh, you’re a tester. That means you just sit and click on buttons and enter the word beer into a numeric field.
Oh, you’re a manager. You simply create, spreadsheets about expenses and fill in performance reviews. And we kind of get this very simplistic view of what the job entails. And what we’re saying here is actually spend your time during this interview reflecting and eliciting what is the actual job. That you need to fulfill, and it may be that you have it really easy and you can just look at people who already are in your organization and you’re say, we want to continue that behavior. So what they’re doing, that’s what we want. Have done this before. You sit and you watch a group as they work and maybe you take notes on what’s going on and that starts becoming a bit of the list of what it is you might be looking for in the interview.
Mon-Chaio: And two things I’ll add to that. One is that some of our listeners might be saying, well, that sounds difficult. I already know how to do a whiteboard interview. I’ve done many of my career. Perhaps it doesn’t take any time. I can just jump in a room and do it, listening, taking notes, designing a whole thing.
Oh, I need to have them solve this problem. That’s similar to the types of problems we’re solving. That means I have to create a whole new problem. I might have to write . You know, some stub code that they’re gonna be working on, and so I have to take some opportunity to do that right off the bat.
Yes, it does take time,
Andrew Parker: Yeah. It does take time. You’re, absolutely right, Mancho. It does take time, and I would say, this is a philosophy that I have about once you’ve decided to hire,
You’ve decided to hire because you realize you do not have enough people to achieve what you want to achieve. You need more people. That means that in order to achieve what you want to achieve, you need to hire. That means hiring has to have a higher priority than all those other things.
So one of the first things to realize as a hiring manager is as soon as you’ve decided that you’re hiring, that’s your primary priority.
Other than stopping things from falling apart, you stop things from falling apart, you do your hiring, and then after all of that is taken care of, then you do all of the stuff to keep another project going because hiring is the thing that gets you going in the future.
Mon-Chaio: I think Andy, that’s a a little bit of a simplistic way to think about it, but you’re not too far off, in my opinion, the way that I would put it. As I would say, hiring is one of the highest leverage things that you can do as a leader and as a company. Two episodes ago we talked about simplistic hiring and hiring decisions and the disservice it makes towards the individuals that come into your organization.
Designing a hiring process is very high leverage or requires deliberate thought. You gotta spend time on it
Andrew Parker: And it does take a long time. Like it
does does
Mon-Chaio: a long time.
Andrew Parker: and it takes a lot of work. Getting hiring right takes a lot of work, but I think it’s worth it. Otherwise I wouldn’t have done it. But I think it’s worth it.
Mon-Chaio: Exactly. Don’t know if everybody would say that it’s worth it. There’s different ways of thinking about the problem. But if you think about it in terms of making sure that there’s the right long-term fit for both the individual and the organization, as the organization grows, I don’t think there’s much more higher leverage stuff that you could do.
So spend your time on it.
Andrew Parker: Yeah.
Which means one thing that we haven’t talked about yet, which is thinking about what you’re hiring for is that long-term fit. This starts touching a little bit on who’s gonna be doing the hiring and mhow, I think we agree on this, but I’ll just check this. It’s the team that that person’s going to be working with should be doing the hiring interviews.
They’re a major part of the entire thing. I would say that, in addition to them being involved in it, that also means the priority of hiring that I gave, and that you kind of said, it’s the high leverage activity, it applies to them as well, which can be hard to swallow sometimes that software
engineers or testers or product managers or whatever are going to be told. Yeah, that meeting that you had scheduled about designing that new system, , this interview, this candidate, they’re coming in at that time, move. The other thing that’s the way it has to go. because they’re trying to find their new team member to help them continue operating.
Mon-Chaio: And here again, it’s context dependent, but much less so than I think many people think. Getting back to this idea of design your hiring process as close as reasonable to the way that you normally work, you are an organization where what you value is more individual work, and you’re moving people every three, or four weeks to a different team, or a different technology or a different project.
If that’s your organization, I think it might be reasonable to do more what you might call panel hiring, where you say, look, any engineer that’s available, we’re gonna test them on some basic skills that we think are transferable and we’ll go from there. It’s certainly more scalable, right? It allows you, no matter which team is hiring, you could use the weight of the entire organization to drive your hiring process.
Andrew Parker: That’s very true and that’s if you do need to hire Aggressively across an entire large organization. You’ll probably end up doing that just for those reasons. But with this trade off of realizing that you now have people coming in to a team where maybe no one on that team had a say in whether or not that’s the person that they wanted.
Mon-Chaio: Mm-hmm. The reason I say that, people think it’s context dependent, but it’s really not. I would challenge anyone who says that that’s the way their organization actually works. To really look, sit down, do that Gemba walk, take those notes and look at how their organization works. Because I would say that in my, you know, over 20 years of experience, greater than 90, probably 95% of organizations do not work that way.
They’re very, very team centric. While they may not say they believe in teams and their performance review process may not incentivize teams, you will see the down in the weeds. People are grouping together in groups of 2, 3, 4 to solve problems. And a
Track 1: lot
Mon-Chaio: of those groups tend to be pretty stable in terms of how they work together.
Therefore, I think we would both say that in general, you should bias hiring process towards. The people that will be working with them are gonna be the people that are gonna be making the decisions, and so they’re gonna be the people that are talking to them. Even if that creates an outsized, what I call an outsized effort or outsized tax on those folks, you know what McKinsey would call the, uh, outer loop from that article
Andrew Parker: the part doesn’t matter. And that according to McKinsey, you should just get rid of if you can.
Mon-Chaio: Exactly. But speaking of getting rid of stuff though, I think maybe this is a good transition into biases. ’cause we can’t talk about designing a hiring process, I don’t think, without really getting into biases. Right.
Andrew Parker: Yes. Absolutely. And it’s really important and an important role for the hiring manager in smaller organizations especially, which is About giving coaching on this, once you’ve figured out what it is you’re looking for. ‘ cause one of the first parts of getting rid of bias is knowing what it is you’re looking for and what it is you’re not.
Mon-Chaio: Mm-hmm.
Andrew Parker: If you are looking for men, you say you’re looking for men, I would not suggest doing that. That would be a terrible idea. So you should say, we are not looking for people who are of a specific gender. You might not say it as explicitly, but you’re going to say that clearly like we are trying to hire diversely or we’re trying to hire in a particular manner.
You call out that you’re trying to do some of these things. And so that is one aspect I consider of the hiring manager is to give that clear guidance on what you’re looking for. And that’s one part of getting rid of bias. And I found this very interesting. I was looking for papers on this mancha, that one way of getting rid of a gender bias, one of the most reliable ways it appears is to just make sure that you are getting a representative of the world population, not of your domains population of candidates.
Mm-hmm.
the programming profession, for instance, is around 80% men, 20% women. That was the most recent number that I saw from one of these papers. But if you only see candidates in that proportion, it reinforces that proportion. And so one of the ways of ensuring that you can hire 50% women, 50% men, is to see candidates that are 50% women, 50% men.
It sounds kind of simple until you think about how are you going to do that, but it was a very interesting take, which is that, and sometimes our bias comes simply from the system that we’re in and we need to just work against that.
Mon-Chaio: That makes a lot of sense. I think tactics on reducing gender bias are a little bit, more than we have time to discuss here.
Andrew Parker: Yeah.
Mon-Chaio: I think that that tactic that you just mentioned is a great sort of signpost to start with. As you think about reducing gender bias, there are other types of biases that creep in also.
And I think one of the big ones that I often think about is historical or concrete skills that you’re testing for versus projection. And it happens both at the IC level and at the leadership level. I’ve been doing a lot more leadership hiring recently, so I might start there.
Often in leadership hiring and in the, what would they call it, the soft skills portion of an IC interview. There’s always one, right?
Andrew Parker: Yeah, there’s always the, sometimes it’s the cultural fit or sometimes it’s the informal chat But, or it might be more explicit. I’ve, seen for senior engineer roles, the expectation that they can give a presentation to describe something they’ve done,
Mon-Chaio: Mm-hmm. , absolutely. In those types of interviews, they often talk about star interviews, and for those that don’t know, we’ll put it in the show notes, but STAR stands for a situation, task action result. And the idea is you ask them to talk about a time when, tell me a time when you’ve done X, Y, or Z. If you think about coding interviews, they’re often like that too.
a proxy for that. It’s tell me a time when you’ve been able to.
Andrew Parker: implement quick,
Mon-Chaio: Show me a time when, right. Exactly. without the, aid of a search engine in the specific language that you’ve chosen. Right. So that’s what you’re testing for. I think star interviews are really interesting. They’re very prevalent these days, I think they run into their own biases if you’re not careful.
And one, I think is the past versus present bias. If you are a listener to this podcast for a long time, you’ve at least heard me talk in some older episodes about problem with leadership and small sample sizes. If you have a leader coming in, they have 20 or 30 years of experience, even a very senior leader.
When you ask them about a specific situation, their sample size is small. If you’ve ever had to manage someone out, okay, let’s say you are a leader that’s had 30 years of experience, how many people have you ever had to manage out directly? 10. Maybe, maybe a little bit more, , or let’s say, how many times have you ever had to take a nascent project and turn it into a billion dollar product?
Three,
Andrew Parker: probably for most people, one or never.
Mon-Chaio: one, or never? So
Andrew Parker: when
Mon-Chaio: you ask a star interview about those types of things, what you’re getting is what they’ve done in the past. And if you don’t do some sort of projection to the future, and if you don’t do the right projection to the future, what you’re doing is you’re biasing towards the tactics that they took.
Versus the skills that they displayed that are transferrable to future problem sets. And I think that’s one of the biggest problems also with coding interviews is you’re saying, look, well, they weren’t able to iterate through this array. Well, they have to be able to iterate through this array. I mean, it’s a stressful situation, right, Andy?
When you’re like coding on a whiteboard and two people maybe are staring at you. So, so this concept of like, yes, star interviews are a little bit more, concrete, more discreet in sort of what you’re evaluating. But this gets back into the what are you looking for? do wanna make sure that you think about how you’re gonna look for those transferable skills.
Not just the ones that you would’ve done it the exact same way in that situation, which a lot of people say, right? They’re like, oh, well, you know, I don’t think we should hire that person. Because in that situation I would’ve gone to the platform team and really negotiated a plan. That person didn’t even go to the platform team, they went over here.
I don’t think that was the right way for them to solve that problem. But what was that transferable skill that you are searching for? And could you have asked another question to see if they had it in a different sense? Do they have a, you know, if you’re saying it’s collaboration, okay, did they show collaboration in another way?
And then is it just a small coaching thing to say, Hey, by the way, make sure to like collaborate the way you do with legal, with your platform teams.
Andrew Parker: Hmm.
I take your, point Mcha, that, , if you ask for has someone done this specific thing, can they give an example of it? You’re biasing towards people who have already done it. And unless that’s exactly what you need, you’re not allowing for , the opportunity of people who have the right approach or, an interesting different approach from what you may be looking for.
I have seen organizations where they say, absolutely we need someone who’s done this before because they don’t have time to coach or to work through someone else learning it. They want someone who’s knows it and can do it. But that’s a very special circumstance where it’s like, this person has done this before. Sticking with the software engineering leadership. It’s like, if you are in a scaling company where like you just got another round of funding and you need to grow your team, that might be the time to hire a manager who has done it before because they
know kind of like, what’s about to happen? I. Even if they’ve only seen it once. But if you’re maybe a little bit more stable or you’ve already got that person, you don’t need to have a manager necessarily who has done it before of going through that, uh, VC funded hyper-growth stage because they’ll have structures around them to help. And so you might be looking for something else.
And if the only thing you asked about was have you done this hypergrowth before? Well, one, you might limit your candidate pool so much that you’re not gonna find them. And two, you’re gonna not get, , a bunch of interesting other ideas of people who haven’t done a before. And so end up coming with a different idea about how to do it.
Mon-Chaio: Right . Some thoughts on that. When you ask about what they’ve done before, it’s the star interview and I apply this to coding questions too. I think your brain automatically goes into, these are the steps I would take. If you’re saying, Hey,
I have a graph and I’m trying to find the min point, you already have an idea in your head how you would go about it, the steps that you would take. And when the candidate doesn’t follow those steps, unless you’ve been very careful about designing that question, removing your own biases, you’re gonna start thinking, well, that’s not the way I would’ve done it.
That’s not the most efficient way. Oh, I read in the book what the most efficient way is, and they’re not taking the most efficient way. That’s a bias in and of itself. Like, are you really looking for the most efficient person or are you looking for the person who’s read that book also and can pa it back?
That answer for you. Similarly in the Star interview, it’s the same thing if you’re saying, oh, have you managed out a person before?
Andrew Parker: You’re
thinking
you are going to be comparing what they say against some sort of expectation that you had. Yeah.
Mon-Chaio: and you’re gonna have to ask those questions, but it’s really important that you think about what is the signal that you’re asking for? Are you really interested in whether they’ve managed somebody out before or is it more, how do they identify low performers or is it more is their coaching style?
And can they relate to people that perform in a different way than them and get them to be high performers? Or is it more, can they look at their team? Identify bottlenecks in performance, whether it be people or systems, and you are never gonna be able to ask all the questions in all the time, and so you have to have some questions where
They’re a little bit more structured, but you do need to be able to understand your biases there. And I think one thing that I would suggest in terms of a tactic for that is a question bank. Andy, have you ever used a question bank in hiring either leadership or ics?
Andrew Parker: I actually haven’t, the closest I’ve gotten to this is A set of people , I had a small team to be working with, but a set of people who I knew what kinds of things that they asked and I’d vetted it and, done, preparation interviews with them. So I knew what it was, but a question bake when you brought this up, I was like, Ooh, that sounds like a nice idea. I like this. So if you could take us through what a question bank is.
Mon-Chaio: will preface this by saying there are companies I’ve worked with, with question banks that have been anywhere from reasonable to not so reasonable, so, Just having a Wiki page full of interview questions is the very minimum to start, and it could be even worse than not having that page. I think the value of the question banks is twofold.
One as a hiring manager or as a craft ative interview loop, you can then say for every question, these are the things that I want to see talked about and vetted in order to make sure A, we’re getting the right signal, and B, we’re getting around our biases. for example, for a coding interview, you have a question bank of coding questions to ask.
But think a very important attribute that needs to be in the question bank for every single coding question is how can the candidate be successful if they don’t arrive at the optimal solution?
Andrew Parker: It goes down to what I think must be a, a very important aspect of this, which is in that question, What is the evaluation criteria and the evaluation? Criteria shouldn’t be that they got Solution X,
Mon-Chaio: Mm-hmm.
Andrew Parker: and unless that solution is so important to that question that if they don’t get it, then it’s all wrong.
But I doubt that’s the way that we’re hiring most roles. And so it’s what is the criteria you’re looking for? What is the kind of thing you’re looking for? Is it that they thought about it? Maybe they didn’t get that point. Maybe they didn’t get that optimal solution, but you could tell that they were thinking about optimization.
Mon-Chaio: Mm-hmm.
It’s funny you mentioned that’s not how we hire for those roles. It actually is. It’s not how we should be hiring.
There’s a large
Track 1: portion
Mon-Chaio: of the world that does hire like that. A company that I worked at with their question bank they had in their, for every level, the approximate time it takes for them to solve that problem historically.
And so you would get into these interview debriefs, which we’ll talk on later. well this was a senior candidate and for most senior candidates, they’ve been able to solve this in 15 minutes. This person was at 20 minutes and didn’t arrive at the solution. So worse than most senior candidates. I don’t think that’s a great use of a question bank.
And so big part of it is as the hiring manager, having a question bank, but making sure the attributes about that question are well vetted by the hiring team and maybe by other hiring teams, if you have a question and you say, okay, well what does it look like to be able to solve this successfully?
Or what does it look like to get positive signal about the candidate without them arriving at the optimal solution? And two people answer that question, that’s not reasonable, right? That discussion is the important part, and the vetting is the important part in making sure all of those questions are vetted so know what is the signal we’re looking for.
Similarly, in the leadership side, having a question bank for leaders, you could start out with a very focused question, which is, have you ever had to manage someone out? But then having the attributes around that, why are you asking that question? What is the signal you’re looking for? And the discussion around, no, no, no.
This isn’t really the signal. It’s actually that signal. And how would you get there? Okay. If they’ve said they’ve never had to manage someone else, where do you go if they said this, but they’ve had a very structured way of managing someone out. What are the . Branching points on how can you elicit your next question to get the right signal.
I think that’s a huge important part of
Andrew Parker: Yeah. This makes me realize that I have on a small scale, worked with a question bank. I wrote one for doing an interview, , and it included, this is the question. This is the, , set of things that we’re looking for this role. This is why this is the context of the company. These are the two questions.
This is the purpose of this one. This is the purpose of this one. These are the evaluation criteria. These are the things that I might not explicitly tell them, but I will prompt them about. , and these are the things that I want to try to get them to bring up. So I had a lot of guidance. It was about two pages long for what ends up being a, 30 minute interview. And then even once I had that, , it was really useful because when I went to give the interview and then when I went to think about what did I think of this interview, it would bring me back from a bias of saying, I had a really nice discussion with this person, and now I could look at this and say, except they didn’t hit any of these criteria.
Mon-Chaio: Mm-hmm.
Andrew Parker: And now I could think about it and be like, yeah, actually they weren’t a great candidate. I liked them as a person. They were very personable. They were , great to talk to. , I probably enjoy working with them, but they’re not what they were looking for for this role.
Mon-Chaio: And getting back to our earlier question, what does a hiring manager do? I think that is a key part of a hiring a manager’s job, to be able to hold the ground on those types of things. I’ve certainly been in scenarios before, especially in these mass hiring waves where you interview people and you’re saying, well go pick a question from the question bank.
And someone comes into the debrief and you’re like, well, I asked this question that wasn’t in the question bank and here’s how they performed. you know, you’re trying to churn these things out and you’re saying, oh, well I have another candidate to go to. Okay, it’s one of six interviewers. He’s got some signal around coding.
I’m moving on. , I think it’s really actually important for the hiring manager to hold a bar and say, no, wait, wait, wait, wait. That’s not from the question bank. , that’s not valid signal. We’re either gonna do it again or we’re gonna discard the signal, or we’re take it on provision, or we’re gonna put you in retraining, or whatever the case may be.
But if you don’t hold the bar, and we’ve talked about this a lot for many different aspects of running an engineering team, once things tend to slip, it’s that broken window theory, right. They start to slip and they start to slip and they start to slip.
Andrew Parker: Normalization of deviance.
Mon-Chaio: Normalization of
Andrew Parker: for any Sydney Decker fans out there.
you know who Sydney Decker is? I, I don’t know if we’ve talked about him. Moncho.
Mon-Chaio: I do not.
Andrew Parker: Oh, he’s, he’s the flying professor in Denmark who, talks about airplane crashes. He’s entirely about just culture and, how to properly handle disaster.
within that discipline, there is something called normalization of deviance, which is exactly what you were talking about, that
slowly over time, because a disaster doesn’t occur, the non-standard, non following procedure behavior becomes normal. That doesn’t mean that that’s the right thing, it just means that nothing terrible has happened yet.
Mon-Chaio: See now I know, just enough to be dangerous. I will use the name Sydnee Decker. It’s funny because, , , my wife, Kay was listening to one of our episodes and she said, oh, I heard Andy mentioned Bandura. Do you know anything about Duda? I was like, no. And she was like, blah, blah, blah. He’s one of my favorite psychologists.
Ah, right. And, uh, I was like, okay, well I guess Andy knows about him. She’s like, he does not
Andrew Parker: I don’t, I think that was from a quote that I found somewhere and it was another paper quoting one of their papers and I, have to admit, I do not know that person very well.
But
Mon-Chaio: to our listeners, we’ll just say, we may quote people, we know a little bit about them, but Albert Bandura is Albert . I think so. , is my wife Kay’s favorite, one of her favorite psychologists.
We don’t know anything about them. Go to her if you need any
Andrew Parker: information.
Uh, back to hiring and software engineering, hiring. I think we’ve now gotten to another key role of the hiring manager, which is, as you were saying, that kind of quality control, quality assurance about the process. And
part of it is making sure that you’re actually following your process. , another part of it is cia.
When you’ve got this bank, do you just throw it over the wall to the people who are gonna be giving the interviews and say, there you go, run that, or do you do something more? This is a leading question.
Mon-Chaio: Yes. I actually just do throw it over the wall the people running the interviews, but, but, but prior to that, I think you probably would agree. And if you don’t, Andy, you can challenge me on this. I am a big believer in training your interviewers.
Andrew Parker: Yes. Yes.
Mon-Chaio: nobody gets to interview at a company until they’ve been trained.
And there’s a number of reasons for that. One is that every company’s culture and interviewing style is different. I’ve gone into companies before where I thought their interview style was kind of bunk, and I knew what I was doing and I was gonna bring my best practices, but it’s not about me.
I think the way I conduct an interview personally might be objectively the best. I don’t think that’s or real, but even if it were real, even if I were objectively the best, it doesn’t help the company as a whole if I’m the outlier all the time because the signal then is not consistent and then there’s bias.
so because of that, I think every interviewer needs to be trained. Now, I don’t know how you train your interviewers, Andy, but I’m a big believer in shadowing.
Andrew Parker: I do shadowing and I do mock interviews.
Mon-Chaio: Tell us more about that.
Andrew Parker: So I have kind of three steps , in preparing the interviewer. , one is , once they’ve got their question, what they’re gonna be talking, , to the person about, , I discuss it with them. I do the kind of like first test on, does it seem like that this is connected to what it is that we’re looking for? And I ask them how they’re going to be evaluating it. This is kind of reaching back to a time before I had this stuff written down anymore. I would actually have it written down. I haven’t done Irene for a little while now, But it would be, , write it down, write down your criteria, write down what you’re gonna be looking for, write down, , what kind of information you’re hoping that they’re gonna talk about, those
kinds of things. , even if it’s coding, what kinds of things in the code are you looking for? Are there code smells? Are there something like that? And then I do a mock interview, which is we get someone else in the company who, hopefully other people in your company would get through your interview process, So you get someone else in the company and you interview them while I’m there watching. I’m shadowing. , and so we run that interview and we find out, can this person do it? do a debrief and we might have to run this again after some coaching, , talking to them about how to talk to the candidate, what to do with questions. , that’s is getting into how to make it a good interview beyond just the question.
Mm-hmm.
uh, all those kinds of things. And then their first actual outing to go talk to a real candidate. I’m still shadowing them because now it’s for real. Now it’s someone they’ve never met before. It’s kind of high stakes. Let’s see how they do. , and then we do a debrief after that. In addition to the debrief that we would do for just the hiring itself, we do probably a debrief about how did that interview go, give any feedback and, and talk about how it went.
, and then at some point, for most people, I find it doesn’t take that many practices or many iterations, but then they’re good to go and you can let them go for a while. I would still listen to how they’re doing, what kind of feedback I’m getting from them. This gets into the what do you do after the interview, , to check kind of like, that’s my quality assurance on, are they still on course for looking for things? But that’s my approach. Mancho, how is your approach, what’s your approach for these kinds of things?
Mon-Chaio: I think it’s very similar. , I tend to have two types of shadowing. , where did I learn this? I might’ve learned this at Uber, and then I was like, this is a really good idea. , this idea that you use, , shadowing and reverse shadowing. So the first thing you do is you have a perspective interview.
Trainee watch as an experienced interviewer does the interview. And then that experienced interviewer goes back and says, Hey, this is what I thought. Here were my inflection points. This is why I asked this question. This is why I went this way. You do that once usually, and then you do the reverse shadowing part where the trainee does the main interview and the experienced interviewer shadows them.
Gives them feedback like your process, Andy, that can go on multiple times. , there needs to be a thumbs up from the trainer that they’re ready to go on their own. , I also have a QA process, it’s a little more formalized than yours. I do think you get a lot of signal in the debriefs, or you had an interesting name for washouts or
Andrew Parker: wash up. I call them wash up
Mon-Chaio: wash ups.
Never ever heard that before.
Andrew Parker: Uh, it might
be British thing. I, I don’t know. Yeah.
Mon-Chaio: that’s what makes this podcast charming. The cultural mixing
pod. Um,
Andrew Parker: S.
Mon-Chaio: why would they be washing up? Are they eating? Um, so you do get signals in the debrief or wash up, but I am a big believer that every, whatever it is, six months or every year, everybody goes through the reverse shadow again.
Andrew Parker: I.
Mon-Chaio: to make sure that, you know, things change and, people can get complacent in the way that they ask questions and company culture changes.
And the group agrees, oh, we’re gonna modify our interview in this way a little bit. Or a question is removed from the question bank, but the person really has asked it for six months and they continue asking it, things like that.
so it’s just good to get them, you know, sort of reassessed .
Andrew Parker: yeah. I agree with all of that. , and I will add it to my practice ’cause I think those are all great ideas. , especially , the reverse shadow and, ’cause I have done that before. I didn’t really ever think of it too much as a part of the process, but I’ve, had it done. When you are in there, to be someone watching the interviewer. A big tip is tell the interviewee what’s going on and make it a little bit of a joke. , make light of that situation. Make sure that they’re not thinking like they are somehow being subjected to a panel interview when that’s not
what’s happening
Mon-Chaio: Mm-hmm. . General best practices are, you mentioned both of them in that, in that example right there. One, make sure that the interviewee’s expectations are set. They should not be surprised at what’s going on, and you should never be like, well, it was an architecture interview, but guess what? Now we have a different section coming up we didn’t tell you about.
It’s a stressful situation. Make sure that they’re as prepared as possible. I know a lot of companies do HR briefings on what are the steps they’re gonna go through and what are the signals they’re looking for at each step. I don’t think that’s cheating. I think that’s fantastic.
In terms of preparation
Andrew Parker: To me it’s the only reasonable way to treat a candidate, to tell them what’s going to happen. I tell them when we’re gonna make a decision, what that kind of decision means. And I’ll tell them when, I’ll tell them about what we’ve gotten out of that. , I tell them all sorts of stuff, because to me that’s also partly a signal of the culture that they’re trying to join.
Mon-Chaio: mm-hmm. . Yeah, there’s one part of this that I’m very hit or miss at. I try to get better, but I’m still not sure what the right balance is. , is giving them real feedback in real time about how they’re doing on the interview. I really don’t like this. Candidates are guessing what signals they should be providing, especially on these open-ended questions.
We’ve talked about things like, tell me a time when you worked closely with product management. It’s such an open-ended question. Are you looking for their collaborative skills? Are you looking for them being able to lead project management , you know, when product management isn’t their weight, are you looking for like, what is it that you’re looking for?
And you can never tell the candidate all of this stuff, but like, as much as possible help them realize, oh, they’re giving signal in the right way. And whether it’s positive or negative. Again, like I said, I haven’t been great at that, but it’s something I’m working on.
Andrew Parker: I think it’s really
difficult because of what we were just talking about. As you’ve got these criteria, you’ve got your initial impression, your immediate impression, sometimes it works out that you’re in the room and you’re just like, I, I don’t really like this. I would be very hesitant about telling the candidate much of anything at that point, because I know that when I step away and I start thinking about it, I could completely change my mind, especially in the wash ups that I was talking to you about, because the way I run them, the entire point is to come to pretty much a consensus. And that means that someone’s mind will change, and I don’t think it’s helpful for the candidate, even though it might feel nice, it’s not helpful for
the candidate give them that immediate thing because it just sets up an, uninformed feedback and you want to give them useful, informed feedback.
Mon-Chaio: I see that and I actually do agree with you. I think that’s a really good point. You also mentioned keeping things light when you were talking about, oh it’s, you know, I’m here to deal with training or whatever the case may be. I think that’s also another great practice, which is make sure you do keep things light with the candidate.
We talked a lot about making sure you’re interviewing the way that, or as close to their working style as possible. It’s very rare that their working style is gonna be some interrogation, some one hour one and a half hour interrogation from somebody. Right.
Andrew Parker: I don’t know
how you macho. I am an interrogating people constantly. That is like my way working
Mon-Chaio: I mean obviously it can’t not be that in some ways ’cause you are trying to elicit information from this person. but do try to keep it light and I would say . To your point, Andy, you don’t wanna give the candidate the wrong signal, but when you are questioning something or something doesn’t feel right, I think it’s okay in sort of a collegial way to say, Hey, don’t know.
I’m not sure you approached that correctly. It feels to me like X
Andrew Parker: versus
Mon-Chaio: this sort of stone phase, just uhhuh. And then what did you do next while writing in your notes, , terrible behavior or whatever.
Andrew Parker: and I think that is crucial as part of interviewing, I often tell people If they’re not going to the thing that you think that they should go to, or if they’re not mentioning the thing that you think they should mention, I would rather that you bring it up to them and ask them about it than just sit there and kind of like wait for them to figure out what’s in your mind.
Because
you, if you want to know about it, because you kind of wanna see how they interact with that concept, sometimes you just need to tell them, Hey, this concept, what do you think about that? Will it help here? And they may know it and they just hadn’t come into their head at that time in that interview, in that stressful situation and so I think sometimes it’s really better for the interviewer to be somewhat helpful to help the person along.
They can take that into account. Like, sorry, I had to help this person 20 times. Alright. But I had to prompt them once. No, go for it. Please do. It also makes it much more, , enjoyable for the person being interviewed, I imagine.
Mon-Chaio: Oh, absolutely. And I can’t tell you the number of times I’ve heard this. Oh, well, I gave them this vague prompt because I have these prompt types of like vague, less vague, specific, and like if I ever get to specific, that means I dinging them three points or. I don’t know. There’s so many bad interviewers out there.
I, yeah, that’s, that’s, that’s a disservice. I would say that there’s bad interview ideas and structures out there that have trained people to interview poorly.
So hopefully what we’ve talked about can help give people more tools. And I’ll hop on this just one last time. I think I’ve mentioned it two times already.
Time, time, time, time, time. It takes a long time to do this.
And if you’re not willing to spend the time and do the right service to bring people in for long-term jobs, just go hire contractors. We’ve said this many times, interview however you want for your contractors. They’re here for a very specific portion of time, and you can correct your mistakes really easily.
And so you’re not willing to spend the time, there are other options out there. To increase your workforce or increase the number of people that can write code or whatnot. But if you’re really gonna hire, spend the time.
Andrew Parker: And I think I’m gonna end us on this. One last thing about the time, and this is the way that I view spending the time on it, interviewing is not just an opportunity for us as the company hiring or us as the team hiring to identify the person we want. It is also the opportunity for the person, the candidate, to identify if we are the company or if we are the team.
That they want to work with if , they want to join. So I consider this all a two-way thing and that everything that we do is about us getting information and about get them getting information. And the time is a portion of that. It’s us showing the willingness to be open about who we are and what we do and how we work. That I think can’t be visible in a one hour interview. So we might do several and we, will have you talk to many different people and we’ll have many different chances for questions to come up or interactions to happen. And so I’m gonna say that that’s the last tactic that I think we should go into that you always view interviewing as a two-way street and take that deep into the way that you think about it, and I think that will make much better interviews
Mon-Chaio: Great. I think that’s a great last tactic to end on.
So we’ll close as always by saying please give us feedback. You can email us at hosts@thettlpodcast.com. That’s multiple hosts since there are two of us. So H O S T Ss hosts@thettlpodcast.com. We are still at the scale that we could read every comment and, , likely respond to everything. So who knows how high touch we’ll be able to be.
But for now, we can be, , we would love to hear from you thoughts on this episode, thoughts on episodes that you would like us to do next, as well as disagreements and things that you think maybe you don’t agree with us and we should think about things a different way. We’re always up for learning.
So as always, until next time, be kind and stay curious.
Leave a Reply