Report from the Breakout Group on Exam Philosophy and Timing
Dr. Dianne McFarlane ? Convener; Fall ACVIM Board Meeting, November 8 & 9, 2002
In attendance: Luis Braz-Ruivo, Deb Zoran, Mary Rose Paradis, Jean Hall, Jane Armstrong, Deb Sellon, Michelle Barton, Ken Hinchcliff, Robert Rosenthal, Joane Parent, and Dianne McFarlane
Following a lively and productive discussion of the philosophy and timing of the ACVIM examination process, the following model was formulated. The model conceived is a single examination session comprised of both a general and specialty examination.
1) The goal of the examination is to identify minimally competent internists in their respective specialty. The group was unanimous that a minimally competent ACVIM specialist should have knowledge in both clinically relevant and basic veterinary science (eg. pathophysiology, physiology etc.) Therefore, the group was in favor of maintaining a general exam, as described. A general examination will be given to all members of the College. The goal of this exam is to test the candidate?s knowledge of physiology, pathophysiology, clinical pharmacology, microbiology, or other topics considered to be core knowledge common to all ACVIM members. There will therefore no longer be a species-specific section to the general exam. The general examination will be comprised of single answer, multiple-choice questions selected from a database of questions validated for both difficulty and content. A sufficiently large database of validated questions will be created to allow this examination to be given two times per year.
2) A specialty (certifying) examination will be given by each specialty. Content of this exam will be based on a blueprint. The blueprint will be designed based on information gathered from Diplomates of the college by a survey. The survey will seek to define what type of cases a boarded internist sees, what activities (imaging, etc) a boarded internist engages in, and what additional knowledge an internist needs to know (e.g., rabies is an infrequent disease but a critical one to have knowledge about.) The survey will therefore be used to determine what a candidate needs to know to be minimally competent in the current market. This survey will be designed to guide both residency training and examination content. The original blueprint was designed when the vast majority of internist worked in an academic environment. With a large proportion of internists now in private practice reevaluation of the residency training, and certification requirements is essential.
3) The specialty examination will no longer be comprised of a recall multiple choice, case based multiple choice, patient management and an essay section. The patient management section (scantron) will no longer be offered after 2003. All specialty examinations will gradually be changed to a completely single answer, multiple-choice test. A significant portion of the exam will be case-based multiple choice. These questions are equal or better than the patient management at assessing problem solving according to the information provided by the ABIM vice president, Lynn Langdon via conference call. The data from the ABIM is based on 10,000- 20,000 tests given annually, over a thirty- year period for which statistics have been collected. All specialties will incorporate their multiple-choice questions into a single database, to facilitate sharing of questions between the specialties. Over time, questions may be validated for both difficulty and content prior to administering them to the candidate. Also, as the databank grows, it will be possible to pre-test questions, by including a number of questions as part of the exam that do not contribute to the score, but are in the test to generate statistical analysis prior to use as an official test question.
The goal of these described changes to the specialty exam are to emphasize a broad scoped, content driven examination, with predominately analytical or problem solving (verses recall) single answer, multiple choice questions. The well written, single answer multiple-choice format is highly reliable, valid and defensible. In addition, it facilitates easy adaptation to a computer- administered exam, which is a long-term goal. Computer administration will allow enhanced images including video as part of the exam. Until computer testing is implemented, a paper- based test, with hard copy images was suggested.
This model allows for moving the examination from the forum to a fall/winter date. This move from the forum may involve only the large animal internal medicine specialty or alternatively, all the specialties. Flexibility in the model will accommodate either.
Advantages to moving the examination away from the forum, to the fall/winter include:
a. Less expense to the college and candidate in administering the examination at a site selected. A location can be selected for its economical hotels and airfare, rather than its ability to house the forum.
b. An exam that is completely multiple choice will only require two or three committee members to administer, rather than the entire committee, as is the case for internal medicine certifying exams.
c. When/if computer administration of the test is employed, there will be more economical options for computer resources by not having it with the forum.
d. LAIM programs lose resident support in clinics in the spring of both the second and third year. Due to the high caseload during foaling season, this is a severe disadvantage to the exam being administered during the forum.
e. If the examination is administered in the fall/ winter, the resident will have the results of the examination prior to leaving a three year program, and ability to award the certificate in the June forum of their third year.
Disadvantages to moving the exam away from the forum include:
a. By not requiring the residents to test at the forum, a drop in overall forum attendance might result. Possible solutions suggested included making a positive reason for residents to attend. One possibility is to offer an exam preparation session. Residents would attend in the second year to prep for the test the following fall. In addition, resident abstracts would likely increase when divorced from the stress of concurrent examination. (Perhaps offering a free admission to the following year?s forum to the winners of the abstracts would be added incentive.) Corporate sponsorship for the residents attending the review session/ forum may also be possible. In this plan the residents attend to the forum (and even to the talks!) the spring of the second year for the review, take the test in the fall, and attend the forum the following spring (their third year) to receive their certificate.
b. If only the large animal group moves to the fall/winter there are potential ACVIM office staff deployment issues. If all the specialties move, staff is better deployed as forum and exam are at two different times. If only LAIM moves it may represent an increase in workload. However, elimination of the patient management and essay sections will serve to decrease office staff workload. As well, a change in case reports, such that they are no longer part of the credentialing process, will greatly facilitate staffs? ability to provide exam support in fall.
c. Case reports can no longer be part of the credentialing process as they require 6 months from the time submitted until review and appeals are finished. It was suggested case reports become a resident training process. The case reports are written by the candidate. The residency mentor then evaluates the report. This plan will allow the case reports to continue to be of value as a learning exercise, with more feed back to the candidates, but eliminate the difficulty associated with use of a subjective evaluation tool.
d. If the exams are no longer administered at the forum, the validation process will need to be done differently. Two optiona are have a selected group of validators to whom individual questions are sent or to pre-validate potential questions at the forum prior to using the questions in a test.
Resources required to adapt this model:
1. A survey to guide the blueprint as to exam content requirements. This would need someone with survey expertise to ensure it is done appropriately, but likely can be modeled after the current NAVLE survey. This is not a job analysis.
2. A testing data base program (cost estimate <10,000)
3. Educating question writers how to construct a case based multiple choice question. This can be done by providing written instructions and having a veterinary science editor available to edit and review questions.
4. Establishing a large general exam database. We are starting with a database of 800 questions, but many need revisions.
5. A lot of questions will have to written/revised in a short amount of time. It was suggested having a parallel committee of trained question writers committed to writing a number of questions of a topic suggested by the exam committee. This parallel committee would not meet in person, therefore do not represent a financial commitment.
The group was unanimous about many critical points, including the overall philosophy, the use of an all multiple choice test, and the need to survey our membership to define what it is the internists need to know. The proposed model will allow the examination to be moved to the fall/winter, if those are the wishes of any given Specialty. It allows the individual specialties more control over the overall difficulty of the species specific questions, which should eliminate any bias against neurology, oncology and cardiology in this section of the general examination in the past. The model also facilitates the adaptation of computer administration in the future. It moves to a more unbiased, better validated and more legally defensible format. Finally, once the changes are adapted, it will likely reduce the overall expense to the college of administering the examination.