How a Leader Made the Interview Process More Collaborative

Angela Riggs, Quality Engineering Manager

As a Quality Engineering Manager at The Zebra, Angela Riggs wanted to improve the interview process. Her goals were to make the interview process less stressful while getting better signals about candidates. We talked about how she accomplished this, how her background as a teacher helped her and how to make interviews more collaborative and less like a test.

Can you give us a bit of background on your career?

I'm Angela Riggs. I'm a quality engineering manager, and I've been in quality engineering for about seven years. Like a lot of folks in this part of the industry, I sort of happened upon quality engineering. I used to be an early childhood teacher and realized I wanted a career change, so I took a developer boot camp and interned at a company out here in Portland, Oregon. When I was ready for a full-time job, they happened to have a quality engineering role open. I said, "That sounds good, let's try it out!" It was interesting to discover that quality engineering was something that I was good at and really enjoyed, and I've been in quality engineering ever since! I started as an IC, made my way through a few different companies gaining experience as an embedded QE and with platform teams. Now I'm in my third career as a manager, most recently at a company called The Zebra out of Austin, Texas.

Was management something that you thought you would get into when you did the career change?

It wasn't. Just like my first role in quality engineering, it came about as an opportunity that I took advantage of. A friend at a creative agency reached out to me because they were wanting to build out their internal quality engineering department, and they were hiring their first QA Manager. He thought it was a good opportunity and something that I could be successful at. I built out the quality engineering department from the ground up, and did a lot of work that I’m really proud of! I gained a lot of experience and  realized that I really enjoyed that sort of people change management, process change management, and all that thoughtful people-first approach that comes with being a good manager. I love it!

I love to ask people because sometimes we can get into roles we didn’t expect. Becoming a COO was a shock to me. Leadership wasn’t something I aimed for. Some people do aim and others like me and I think you, they find their way.

Yes, exactly!

You recently left The Zebra, but we want to talk about the changes you made to the interview process while you were there. Can you share some background about the context and what you were trying to solve?

There were two challenges that I wanted to solve. First, interviewing is a stressful experience for a lot of people. And when we're stressed, our brains work against us - the amygdala kicks in and triggers the fight-or-flight response. Your higher functions stop working, which means you lose critical thinking, communication, and recall. All the things that are important to be good in an interview go away, and it's really hard! As a candidate, I get super stressed and I know that's true for a lot of people. So I wanted to reduce the stress of interviewing.

I also wanted to do a better job of setting expectations appropriately to help candidates understand what success looked like for them. This would help us as interviewers do a better job of evaluating candidates in a more meaningful way, with behavioral panels being able to ask more relevant follow-up questions or have a more meaningful conversation about someone's experience. It would just give candidates an idea of what signals we're looking for so they can make sure they're speaking to those things.

What made you want to do that? What was the impetus?

As a hiring manager, changing the interview experience was something that I could do to make a difference for people. I wanted to make the process better. I was also spending a lot of time in the behavioral interviews asking follow-up questions because I wasn't getting signals. Again, communication abilities and recall go down under stress, so it can be harder to extrapolate meaning from people's responses. With our technical interview as well, we weren't setting expectations so it was harder to evaluate candidates. Because we weren't telling them what we wanted, we weren't getting the things that we wanted. This means that it was hard to get the signals that were important to us, so I wanted to make the process easier on our side as well.

That adds friction to your process by adding brain load by having to ask more questions or even more interviews. I also assume that you worry about overlooking someone who was a good fit but you didn’t recognize them because the process wasn’t supporting them to bring out those qualities.

Exactly.

What made you say, "We're going to take this on now"? Changing the interview process is not a small thing, it takes effort.

Right! I had an open position and was in the middle of hiring. I was going through the process and hitting the snags with that process and thinking, “I can do better than this. I can make a process that's better for candidates and us as interviewers." So I decided to make that change.

Where were you in your hiring journey in your career? Had you done much hiring as a hiring manager?

I would say I'd done a lot of interviews, but the bulk of my hiring manager experience was at this current company. So this was really my first time hiring for multiple roles and really going through the process a lot more than in my previous role as a manager.

For context, how big was The Zebra at the time?

Probably about 300 to 350 people.

So at 300 to 350 people, you've got a mature HR department and hiring processes. Were these changes engineering-wide or localized to your team?

It was localized to my team. I partnered with my recruiter who was amazing and open to my ideas about the hiring process.

Sometimes there can be friction with HR because they want to standardize things across a company as they grow. But it sounds like they were open to adjusting the process.

Yeah. There was a nice bit of autonomy here to be able to try something out for myself and my process without it having to be widely accepted.

Who did you have to get buy-in from?

In hindsight, I didn't really need buy-in from anyone. When I brought it up, it was like, "Yeah, give it a try." There was like a lot of trust in, "Hey, I like what you're doing. Try it out and see what happens." I had talked to my VP about it because he had been doing a proof of concept for written interview responses. Instead of starting with the Q&A in person, he started with written responses, and those responses informed the actual face-to-face interview. I love that he was thinking about how to make his interviews more collaborative, and more inclusive! He had a proof of concept doc that he was working on. There were some pros and cons - some people thought that it just added a lot of overhead even though it let them think more clearly about the things they wanted to say. Others who were not native English speakers mentioned that it was harder for them to get their thoughts down in writing. 

On the candidate side, I've been in an interview process a few times where there are written questions in advance and that's a ton of work. When I write emails and communications, I'm thinking, "How many exclamation points did I use? Where are my emojis? What do I need to pull out?" For an interview where I'm trying to get a job, double or triple that mental labor of not just giving my experience, but trying to phrase it with wording that resonates with the people who are reading it. Written communication is so hard! There's no tone. There's no emotion or enthusiasm in the voice. And it was more stressful. I liked his idea but wanted to reduce the overhead for the candidates and try something new.

After considering all the pros and cons, what did you decide to do?

I started with our technical assessment, which was a take-home assignment, and I brought my teacher's eye to it. If you're handing out an assignment, you want to make sure you're setting expectations. You want to make sure your rubric is clear. I realized that we weren't setting expectations well. For instance, we wanted to look at someone's current knowledge and current experience with automation. But we weren't asking for that, and so we weren't always getting that. 

Some folks would complete the assessment in a framework that was new to them because they wanted to show the skill of how they ramp up on a new thing. That's a good skill to have - but it's not the one we were optimizing for. We weren't asking for what we wanted and sometimes we weren't getting it. And our rubric was kind of messy. It was duplicative. It wasn't asking interviewers to evaluate for the things we cared about, and it didn't align with the things that we wanted from the candidate’s assessment.

So it was about updating the rubric to be clearer both for candidates and interviewers. So when we introduced the take-home assessment we said, "We're looking for documentation, clarity of code, and test architecture. And then those expectations that we sent to the candidates were reflected in the rubric that the interviewers used. So not only do the candidates know what the expectations are, the interviewers know what the expectations are. So there was a lot of revamping and adding clarity.

With the behavioral panels, which is the conversation and the Q&A, it was sending those questions in advance. So we had a behavioral panel and then a wrap-up panel with myself and my manager. We had a process where we would all put our questions into a spreadsheet ahead of time to make sure we weren't duplicating questions and that we were aiming for the right things. And when the recruiter sent the scheduling email to candidates, it just included, "Your panel at 11 o'clock is with this person and that person, and here are the questions they're going to ask." We were upfront about it.

What did you notice when you did that? Was it iterative, did you make changes as you went along, or was it set right away?

With the take-home assessment, it was like a one-and-done. I went over it with a fine-tooth comb, we did a lot of editing out making it a lot more clear. It was pretty successful immediately. With the sending questions in advance, it was a bit more iterative. One of the things I realized is that, again, setting context, I hadn't told candidates why I was sending them questions in advance. We had a couple of interviews where it felt like the candidate had done a ton of prep. They had written down answers in advance and were sort of reading from that. So the initial answers were good, but it meant that with follow-up questions, they hadn't prepared as much, and they sort of lost their way.

So, I iterated on that. Now when we sent the candidates that email, we said, "Hey, we're sending you this in advance to help you prime your brain. We don't expect you to write out full answers in advance." To set that expectation of, "We're not trying to make this harder for you." Because that again goes back to the written responses that add a ton of overhead. So really just reminding myself to be clear and set those expectations at all the steps.

I send prep questions for this series. I also say, "This is just a starting point to get your brain thinking about it, not a script." I found that it helped people think about their responses. This made them more comfortable so they can show off their personality rather than thinking about the answer.

Yes, totally. I find the same with public speaking as well. If you over-prep it comes off as scripted, but there's that happy medium of just enough prep to where you know what you're saying, but you can riff on it as needed throughout the conversation.

It also strikes me that in doing this, you're indicating, "Here's how our relationship might go." Setting expectations about your work style.

Part of my job as a manager is setting clear expectations and setting context for people. This is someone's first interaction with our company, with me as a potential manager. Their first day at work is not the starting point. Their first interview, their first email with us is where their impression gets formed and how they start building that relationship. 

We've all been at companies where the interview feels like a battle and it feels awkward and uncomfortable. How does that transition to a really good working environment afterward? It is easy to forget that during the interview process you’re already transmitting a lot about your culture and what they can expect working with you to be like.

Yes, exactly. Trust takes time to build. The earlier you can start the better.

Was that part of the thinking? As a way to start building trust and the relationship?

It really was. This is somebody's first impression, right? It’s my first impression of them but also their first impression of me. I had an explicit goal to try and make that as positive as possible. Interviews don't always mirror what it's like to work at a company. There are often two different approaches, which I think is part of the problem as an industry with our interviewing process. I wanted to make those aligned as closely as possible.

You noticed the shift in the technical part right away. What was the evidence that showed you it was the right way to go?

Previously in our post-interview conversations around the technical interview, we were filling in a lot of gaps for ourselves and having to extrapolate a lot from the signals we did get into whether they were the signals we wanted. So from the interviewer's side, there's a lot less overhead. We told candidates “Here are the tools we use. If you're familiar, use these tools.” We wanted to make it easier for interviewers. Someone had submitted an assignment in Java, which we don't use and which we're not set up for. It took one of our interviewers three hours to set up her environment to be able to run the tests, just to evaluate that candidate. So in the process and the overhead, and also the conversations we had after, they were a lot clearer in being able to not only evaluate the candidates but also cross-compare. Because they're all sort of doing the same thing now, and we can be a little fairer and more equitable in that evaluation.

Once we had the new process in place and asked for the things we wanted, it meant that we could talk about those signals and go, "Here's where we got it, and here's where we didn't." The feedback from interviewers was a lot more meaningful.

Did you feel like it reduced stress? I don't know if you could see that maybe in that process.

Yeah. It's hard to get that signal on whether it reduced stress for the candidates. It probably made the interviews a little bit less stressful. Because if they took one approach to their take-home and then were asked about totally different things in the face-to-face technical interview, that mismatch can be super stressful because then someone's thinking, "Oh my gosh. Holy shit. I didn't know that this is what they wanted. I didn't do any of that!" Making an assumption, I would assume that having that alignment between the take-home and the technical interview probably felt a little bit better for our candidates.

Was it take-home before?

It was always take-home - that wasn’t part of the changes I made. In the last few months that I was with The Zebra, we shifted to an in-person paired assessment instead of a take-home. When we shifted, I still went through the process and made sure we were setting the expectations for the candidate - again, with the teachers' view. Not just verbal instructions, but having written bullet points that they could refer back to reorient themselves. And again, making sure the rubric followed along with the things that we were talking about in the process.

What made you all move to in-person technical paired versus the take-home?

We had both embedded quality engineers and we had an SDET team. And from the SDET hiring, they were losing a lot more candidates because of the take-home. There are benefits and trade-offs to either type of technical interview. They were noticing more of a drop-off in the take-home assessment. So they decided to try the in-person assessment instead, which also aligned a bit more with the rest of the engineering department.

In-person technical interviews benefit people who operate better under solo stress, and then take-home assessments tend to benefit people who have more time. So there are disadvantages to either one. Ideally, I would love to work at a company that has the ability and capabilities of offering candidates a choice. I'd love to be able to say, "Hey, do you want to a take-home or do you want it to be in person for the assessment?" But I think that can be asking a lot because that takes a lot of time and effort. People and companies tend to not think about the interview process until they're doing it, and that requires a lot more planning.

Yeah, there are a lot of organizational pieces to move. I would love that too because some people also don't have the time to take something home and to feel like I can devote the time to do that. And you might be missing people who are great. Single parents, or other people who are working multiple jobs to make a living, whatever it is.

Exactly. Yeah.

How did you go about making changes? Was it you? The hiring committee?

It was both. They also felt like they weren't able to get the signals they were looking for. When it came to changes, it was just me communicating with the recruiter and with the folks on the interview panels to say, "This is the process we're doing. Please give me feedback on it.” And then taking that feedback and iterating when we realize like, "Oh, we need to be better at setting context for why we're doing this."

It’s interesting to figure out how to make these decisions because being involved in changing the process might be overwhelming for people who are interviewing on the side of the day-to-day work.

Exactly. Yeah. I'll dive into that a little because I do think interviewing is part of the job and it's not for everybody. Not everyone is good at it. Not all companies take the time to train people. We all want great coworkers, so we all have to do the work to get there. But there were challenges to the changes I was making. A couple of people felt like it was cheating to send people the questions in advance. I think that speaks to one of the ways that when we say, "Interviewing is broken." I know that that's a way that I've grown as a leader, is how I approach interviews. I used to think of them as a test to sort out good candidates from bad candidates, which is a common mindset. But now I realize that it's not the right thing to optimize for. 

Candidates and employees are not the same thing. A good employee is so much harder to get a signal for, and so we often use who's a good candidate as a proxy for that. But a good candidate is someone who can do well under pressure on their own and someone who can remain confident and cool in that specific situation. As an employee, you want someone who can look up the things they don't know, ask questions, and collaborate. One of the things that I've come around to is interviewing shouldn't be a test. We shouldn't be looking for gotchas. We should be trying to assess whether someone's a good employee, not a great candidate. 

And again, going back to, "Is the interviewing process reflective of the work environment?" And treating it as a test ideally is not reflective of the work environment.

You make a great distinction between someone who can interview well versus someone who is a good employee. Some people do perceive interviewing as a side job and at some companies, it’s treated that way. It sounds like it's built into the culture at The Zebra.

Yes. And you're totally right that it has to be a company-wide thing for people to view it as part of the job and not a thing that detracts from the job. We had an amazing recruiting team. Our whole team of recruiters was very inclusion-focused and super empathetic. It was important to them to have a good process. It was great being able to work with them and make that a priority for my hiring process as well.

There’s a cultural and mindset shift needed to make these changes. How did you get folks on board?

It was really just having those conversations and explaining my framing. Showing the work of why I didn't see it as a testing situation, and why I wanted it to be a collaborative relationship-building conversation. Instead of just throwing it out like, "Here's what we're doing." It's like, "Here's what we're doing, and here's why we're doing it." It took some conversations. And not everyone changed their view to agreeing with me, which is totally fine. But they still participated in the process that I was setting and went into it with good intentions, which I think is important. That ability to say, "I don't agree with this. But at the end of the day, as hiring managers, it's your process, and I'll go into it with good faith," I think made a big difference.

That's interesting too. Not everybody changed their mind and they decided, "I'm going to disagree and commit. I'm going to move forward with this."

Yes. I love it because I think “disagree and commit” sometimes gets misused a little bit. Some people think that it means you can keep disagreeing the whole time that you're doing the thing. And it's like, No, it means you've shared your opinion. It's been heard, and a decision got made. So you're putting all your efforts into that decision, and not continually trying to pull it back. It made a big difference that people went in with that ability to separate their opinion from the process that they were using. It was super helpful.

I can see the influence of being a teacher. The way you think about how people process information and the way they go through stressful events. It reminds me of school where some might understand the information but don’t test well. Is there similar learning there for you around the people who interview well versus people who might be the right employee?

Yeah, exactly. Some kids know how to learn, they know how to process things. They're learning about things. They're reading. But under the stress of “pass or fail”, they just can't. Fight or flight kicks in, the brain shuts down. It just feels terrible. I sort of joke that as adults we don't really change that much. We're all just taller four-year-olds with bills to pay. A lot of the same approach to people and thinking about people applies whether it's a four-year-old, whether it's a 40-year-old. How you allow people to process, how you give context, how you carry out change management — all those things don't change all that much! The situations you're applying them to look different, but the actual approaches are often very similar.

The people who weren't able to shift their mindset but went forward anyway, were there patterns you noticed?

Yeah. It was really around the purpose of an interview. The view is that it’s a test, to separate out good people and bad people. The ones who fail are bad candidates, or the ones who pass are good candidates. Then there were a couple of other concerns. My manager was concerned about false positives around how candidates communicate. So if they prep, they might seem like they're better at communicating than if they didn't have time to prep. I understand the perspective, but it's not one that I agree with - no one's getting a pop quiz at work. If you're asking them questions, they might know the answer already and they'll give it to you or they might go, "I don't know, let me go look that up, and then we can talk about it." So it was just, again, the approach to why are we doing interviews? What outcomes do we want? It really came down to how people view the purpose of interviewing and what they think it's for.

That makes sense. How was it persuading your manager and getting them on board with where you want to go?

Pretty easy. I just said, "Hey, here's the thing I want to try." He didn't necessarily agree with all of it, but we had a lot of trust. So he gave me the freedom to implement it and see how it worked out.

So even though he said, "I don't necessarily agree. This is not the approach I would take." he was hands-off. He didn't say, "I'm not sure this is a good idea."

Exactly. It's still going to be my decision.

It sounds like you were doing real-time analysis. Did you do a postmortem afterward? What did you learn?

It's tough because if someone has succeeded throughout the hiring process that means they're an employee now. Obviously, they're a little bit biased that this was a successful process! It works for them, right? And if someone doesn't make it through the process, it may not reflect as a good process for them. So it can be a bit hard to get meaningful feedback. I still asked about it because it's still useful to know if there were any pain points or gaps along the way. It was a pretty positive response. People felt that the technical assessment was pretty clear and they liked getting the behavioral questions in advance. And then from the interviewer's side, getting the feedback was a simpler process. It was easier to get the signals they wanted and to evaluate the candidates on those signals both from the behavioral point of view and from the technical assessment.

Did you run this process again? Did you make any changes for future iterations?

Yeah. I was lucky enough to find that fit with that first role that I was hiring for. The process went on long enough that I could see where it was working well. So when new roles opened up, it was just applying the same process.

That's fantastic. I'm just curious, did any other parts of the organization adopt any of this or borrow from you?

Yeah! A few months later, our front-end engineering folks were revamping their technical interview. They were also doing a take-home assessment and it was a lot more complex. They were looking to streamline it and make it a shorter process for people so it was easier for candidates. So I worked with them andI did a little bit of coaching. When you're sharing this technical assessment, whether it's a take-home or in-person interview, what questions are you asking of the candidate so that they can complete it? What does your rubric look like? Are you evaluating things that the candidate knows to expect, and did you share those expectations with them? It felt really cool to be able to partner with them and help them improve their process as well.

That's fantastic that others were able to learn from that. What advice might you have for someone who wants to undertake a process like this, or has similar goals?

I think there are two parts. One is relevant to the hiring process and the other is more general. First, people have really strong feelings about the hiring process. It's easy to get into the mindset of, "There's one right way to do it. If only we could find it." But there's not, right? This is a really common part of the industry. It happens all the time in our jobs. But we don't put enough thought into it. The effort of making changes and iterating the hiring process was not that huge. Small changes can have big returns to making a more people-centered process.

My initial goal in changing the process was to have it be more collaborative and less stressful. The signals and thinking about getting better meaning from people's responses was a secondary impact. I think it's hard to go wrong when you put people first and act in an empathetic way.

The other thing is what to do if something is not working. If there are gaps in your process, think about the outcomes first. What outcomes are you aiming for that you're not getting? Then think about how the process either helps or hinders that. It’s easy to think about just the process itself like, "Oh, this process is painful." This was a good learning experience for me. I knew things needed to change but it was hard to pinpoint why, especially with the technical assessment. So I thought about what outcomes we needed and then why our current process wasn’t working. Think about the end first and then how to work your way there.

Anything else we haven’t talked about that’s important?

Interviewing is part of your work culture. Interviewing is part of your relationship-building, especially from the hiring manager's perspective. This is your chance to create the culture and relationships that you want to see play out in your work culture day-to-day. So if you start with candidates, give them this culture, give them this trust and safety, and they can help perpetuate that culture once they're also working side-by-side with you. Have your hiring process be reflective of the culture that you want to create.

Here’s an in-depth article about the changes she made.

If this piece resonated with you, please let me know and give the heart button below a tap.