The promise and challenge of including multimedia items in medical licensure examinations: Some insights from an empirical trial

Linjun Shen, Feiming Li, Roberta Wattleworth, Frank Filipetto

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

BACKGROUND: The Comprehensive Osteopathic Medical Licensing Examination conducted a trial of multimedia items in the 2008-2009 Level 3 testing cycle to determine (1) if multimedia items were able to test additional elements of medical knowledge and skills and (2) how to develop effective multimedia items. METHOD: Forty-four content-matched multimedia and text multiple-choice items were randomly delivered to Level 3 candidates. Logistic regression and paired-samples t tests were used for pairwise and group-level comparisons, respectively. RESULTS: Nine pairs showed significant differences in either difficulty or/and discrimination. Content analysis found that, if text narrations were less direct, multimedia materials could make items easier. When textbook terminologies were replaced by multimedia presentations, multimedia items could become more difficult. Moreover, a multimedia item was found not uniformly difficult for candidates at different ability levels, possibly because multimedia and text items tested different elements of a same concept. CONCLUSIONS: Multimedia items may be capable of measuring some constructs different from what text items can measure. Effective multimedia items with reasonable psychometric properties can be intentionally developed.

Original languageEnglish
Pages (from-to)S56-S59
JournalAcademic Medicine
Volume85
Issue number10 SUPPL.
DOIs
StatePublished - Oct 2010

Fingerprint

medical examination
multimedia
candidacy
narration
psychometrics
technical language
textbook
content analysis
discrimination
logistics

Cite this

@article{08f215cfdbf6410585092a97abb3c05a,
title = "The promise and challenge of including multimedia items in medical licensure examinations: Some insights from an empirical trial",
abstract = "BACKGROUND: The Comprehensive Osteopathic Medical Licensing Examination conducted a trial of multimedia items in the 2008-2009 Level 3 testing cycle to determine (1) if multimedia items were able to test additional elements of medical knowledge and skills and (2) how to develop effective multimedia items. METHOD: Forty-four content-matched multimedia and text multiple-choice items were randomly delivered to Level 3 candidates. Logistic regression and paired-samples t tests were used for pairwise and group-level comparisons, respectively. RESULTS: Nine pairs showed significant differences in either difficulty or/and discrimination. Content analysis found that, if text narrations were less direct, multimedia materials could make items easier. When textbook terminologies were replaced by multimedia presentations, multimedia items could become more difficult. Moreover, a multimedia item was found not uniformly difficult for candidates at different ability levels, possibly because multimedia and text items tested different elements of a same concept. CONCLUSIONS: Multimedia items may be capable of measuring some constructs different from what text items can measure. Effective multimedia items with reasonable psychometric properties can be intentionally developed.",
author = "Linjun Shen and Feiming Li and Roberta Wattleworth and Frank Filipetto",
year = "2010",
month = "10",
doi = "10.1097/ACM.0b013e3181ed3c80",
language = "English",
volume = "85",
pages = "S56--S59",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins Ltd.",
number = "10 SUPPL.",

}

The promise and challenge of including multimedia items in medical licensure examinations : Some insights from an empirical trial. / Shen, Linjun; Li, Feiming; Wattleworth, Roberta; Filipetto, Frank.

In: Academic Medicine, Vol. 85, No. 10 SUPPL., 10.2010, p. S56-S59.

Research output: Contribution to journalArticle

TY - JOUR

T1 - The promise and challenge of including multimedia items in medical licensure examinations

T2 - Some insights from an empirical trial

AU - Shen, Linjun

AU - Li, Feiming

AU - Wattleworth, Roberta

AU - Filipetto, Frank

PY - 2010/10

Y1 - 2010/10

N2 - BACKGROUND: The Comprehensive Osteopathic Medical Licensing Examination conducted a trial of multimedia items in the 2008-2009 Level 3 testing cycle to determine (1) if multimedia items were able to test additional elements of medical knowledge and skills and (2) how to develop effective multimedia items. METHOD: Forty-four content-matched multimedia and text multiple-choice items were randomly delivered to Level 3 candidates. Logistic regression and paired-samples t tests were used for pairwise and group-level comparisons, respectively. RESULTS: Nine pairs showed significant differences in either difficulty or/and discrimination. Content analysis found that, if text narrations were less direct, multimedia materials could make items easier. When textbook terminologies were replaced by multimedia presentations, multimedia items could become more difficult. Moreover, a multimedia item was found not uniformly difficult for candidates at different ability levels, possibly because multimedia and text items tested different elements of a same concept. CONCLUSIONS: Multimedia items may be capable of measuring some constructs different from what text items can measure. Effective multimedia items with reasonable psychometric properties can be intentionally developed.

AB - BACKGROUND: The Comprehensive Osteopathic Medical Licensing Examination conducted a trial of multimedia items in the 2008-2009 Level 3 testing cycle to determine (1) if multimedia items were able to test additional elements of medical knowledge and skills and (2) how to develop effective multimedia items. METHOD: Forty-four content-matched multimedia and text multiple-choice items were randomly delivered to Level 3 candidates. Logistic regression and paired-samples t tests were used for pairwise and group-level comparisons, respectively. RESULTS: Nine pairs showed significant differences in either difficulty or/and discrimination. Content analysis found that, if text narrations were less direct, multimedia materials could make items easier. When textbook terminologies were replaced by multimedia presentations, multimedia items could become more difficult. Moreover, a multimedia item was found not uniformly difficult for candidates at different ability levels, possibly because multimedia and text items tested different elements of a same concept. CONCLUSIONS: Multimedia items may be capable of measuring some constructs different from what text items can measure. Effective multimedia items with reasonable psychometric properties can be intentionally developed.

UR - http://www.scopus.com/inward/record.url?scp=77958506271&partnerID=8YFLogxK

U2 - 10.1097/ACM.0b013e3181ed3c80

DO - 10.1097/ACM.0b013e3181ed3c80

M3 - Article

C2 - 20881705

AN - SCOPUS:77958506271

VL - 85

SP - S56-S59

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 10 SUPPL.

ER -