A new video tool helps residency program directors screen applicants

The Standardized Video Interview measures interpersonal communication skills and professionalism, and offers a chance to further assess candidates before in-person interviews.
None

Every year, residency programs across the United States are inundated with hundreds of applications for each available slot. What’s more, medical students in some specialties submit dozens of applications in the hopes of securing a match.

The volume of applications—plus the hours spent wading through submitted materials, reading letters of recommendation, and meeting with candidates—means residency directors are eager for help in quickly and effectively identifying applicants who might suit their programs.

That help could come from an innovative new tool, the AAMC Standardized Video Interview (SVI), an addition to the Electronic Residency Application Service (ERAS)® process. The SVI launched in the 2018 application cycle as part a two-year pilot project among emergency medicine residency programs.

“Everyone is struggling with the increasing number of applications. Our goal is to help applicants apply more strategically and programs select more strategically.”

Atul Grover, MD, PhD
AAMC executive vice president

A one-way, online interview, the SVI asks questions to measure the applicant’s interpersonal and communication skills and knowledge of professionalism. Those areas “are the hardest to remediate, and cause the most problems on a day-to-day basis,” notes Steven B. Bird, MD, emergency medicine residency director at the University of Massachusetts (UMass) Medical School and a member of the working group that advises the AAMC on the SVI.

The SVI is meant to help residency programs narrow the pool of potential in-person interviewees and to help applicants paint a fuller picture of their strengths, explains Atul Grover, MD, PhD, AAMC executive vice president. “It offers a deeper and broader view of prospective residents than do grades and test scores alone,” he notes.

The SVI is the product of a broad effort—medical educators, health care leaders, and members of the residency community all contributed—and is part of the AAMC's Transition to Residency initiative. It’s considered an expected part of the ERAS® application for all residents hoping to enter an ACGME-accredited emergency medicine program accredited by the Accreditation Council for Graduate Medical Education.

AAMC leaders chose emergency medicine for the pilot in part because they deemed its relatively small size more manageable. About 80 percent of accredited emergency medicine programs, or 172, enrolled in the 2018 pilot. The AAMC is considering extending the SVI to other specialties in the future.

“Everyone is struggling with the increasing number of applications,” Grover notes. “That’s more stressful for applicants who are competing for interviews—or who are getting more interviews to travel to and pay for that may not end up a good fit—but it’s also stressful and costly for program directors.”

“Our goal is to help applicants apply more strategically and programs select more strategically,” he says.

Preparing for the SVI

The SVI asks six questions about past experiences and knowledge as well as hypothetical situations applicants might face in the future. (See sidebar.) For interpersonal and communication skills, questions cover topics such as emotional intelligence, teamwork, and leadership. Questions related to knowledge of professionalism look at empathy, cultural competence, conscientiousness, and more.

Monday, July 16, 2018

What does the SVI ask?

SVI questions cover a range of topics related to professionalism and interpersonal and communication skills. Here’s one example:

“Describe a time you worked with a challenging patient. What was the situation? What specific actions did you take? What was the outcome?”

Applicants have up to 30 seconds to read each question and up to three minutes to record their response. An applicant preparation guide details how to practice through mock interviews and with the online SVI platform. Students with disabilities can apply to take the SVI with accommodations.

Some medical schools have been helping students prepare. At the Brody School of Medicine at East Carolina University (ECU), experts in emergency medicine, academic affairs, career development, and speech and communications developed sample questions based on ones the AAMC provides. Applicants practiced through an online interview service. “We wanted to create a scenario that felt real to them,” says Susan Schmidt, MD, the school’s associate dean for student affairs.

The ECU experts watched the students’ practice video interviews and met individually with them to suggest ways to improve.

Zachary S. Williams, a fourth-year ECU student, prepared for the interview with AAMC materials and then took his school's practice interview. Williams says the preparation gave him confidence, taught him how to speak to the camera, and provided a sense of what to expect from the questions.

The feedback he received from the ECU team “helped immensely,” he says. “They went over everything from what we wore to our eye contact, how long we took to respond, and the content of our answers.”

Like some other schools, ECU offered a space where students could be assured privacy and quiet for the SVI. They also provided a strong internet connection to remove worry about interruptions in coverage.

ECU students did well on the SVI, says Schmidt, but if someone does poorly, she says she might advise the student to apply to additional programs or explore other ways to become more competitive.

The SVI doesn’t require extensive preparation, according to Dana Dunleavy, PhD, director of the AAMC’s admissions and selection research and development program. To measure the effects of preparing, the AAMC surveyed applicants and compared their preparation time and strategy to their SVI scores, she explains. The research provided several insights, including that, on average, students who prepared for two hours did as well as those who prepared for eight hours. People who didn’t prepare did slightly worse than other applicants.

Effectively identifying candidates

For the 2018 ERAS application cycle, 54% of participating residency programs used the SVI as a component of their selection process. The other programs received scores but didn’t formally factor them into decision-making. All SVI scores are generated by AAMC-trained reviewers.

Bird says that at UMass, the scores were helpful in identifying candidates for in-person interviews. “If students were borderline [academically] but had exceptional SVI scores . . . then we used the SVI to select those students for interviews,” he notes. He says he values knowing that a candidate has great interpersonal skills and a solid understanding of professionalism.

Monday, July 16, 2018

The SVI rating system

Each question on the SVI receives a score from 1 to 6; total scores can range from 6 to 30. In the first year of the AAMC SVI pilot, applicants had a mean score of 19.

SVI raters are human relations professionals who receive eight hours of AAMC training. Topics covered include how to evaluate the video interviews, the emergency medicine resident’s job, and setting aside personal biases. They also complete eight hours of practice ratings.

To help reduce the risk of unconscious bias, a different rater scores each question, explains Dana Dunleavy, PhD, director of the AAMC’s admissions and selection research and development program. AAMC research shows that scores do not disadvantage applicants by gender, race, or ethnicity, she notes.

“SVI was not time-consuming for us,” Bird adds, “because, most of the time, we relied simply on the score provided” rather than watching the videos.

The AAMC collected evidence to support the SVI's reliability and validity, says Dunleavy. In fact, the AAMC tested the SVI on volunteers before launching the pilot program, and then updated it based on study results and feedback from key stakeholders.

Now, the AAMC is collaborating with researchers in emergency medicine to examine how SVI scores align with ratings from emergency medicine’s standardized letter of recommendation. They are also following residents at 17 programs to assess whether their SVI scores correlate with how well they perform in their residencies.

Dunleavy predicts the results will provide important insights. “We believe the SVI measures factors that aren’t measured as well with other methods,” she says. “Applicants who have a jagged profile with great academic abilities but poor communication skills or professionalism are probably not in the best interest of patients, the training program, or future colleagues.”