Computer Adaptive Tests

Since CAT exams are delivered in a completely different manner than pencil-paper exams, they will “feel” more difficult. Candidates should not be concerned about the ability level of an item on the exam because their ability is being ‘measured’ in a different manner. This works by placing all items on a standard scale in order to identify where the candidate falls within the scale. As a result, candidates should answer all items to the best of their ability. Let’s use an example to explain this:

Suppose that a middle-school athlete is trying out to be a member of the high jump team of the track team. The coach, after many years of experience as a middle-school coach, knows that in order to score any points at a middle-school track meet, his individual jumpers need to jump over a bar placed at four feet above the ground. This is the “competency” standard. If he enters jumpers who can jump three feet, he knows these jumpers will rarely-- if ever--score points for his team during a track meet. Those who jump four feet on the first day of try-outs, after training and coaching, can not only jump four feet (the minimum) but; later may, through additional education, learn to jump five or more feet. The coach knows that it will be worth his time and effort to coach these try-out jumpers to greater heights. Therefore, he tells those who jump over four feet at try-outs that they are members of the high jump team (because they have met the entry-level or competency standard).

Since the coach knows the competency standard, he can hold a try-out to see who meets the entry-level competency. The coach will likely set the bar at or near 3 feet 6 inches for the first jump attempt. Those who make it over this bar will then progress to perhaps 3 feet 9 inches to test their ability at that height. After a group has passed 3 feet 9 inches the coach will again raise the bar to 4 feet and have the successful jumpers attempt to clear it., A smart coach will likely not tell the team the necessary height so that he can learn the maximum ability of each try-out jumper. At the 4 foot level, the coach may find that seven of ten athletes clear the bar. He will then raise the bar to 4 feet 3 inches and later to 4 feet 6 inches. He will increase the height of the bar until he determines the maximum individual ability of each try-out jumper. If he has four slots on his team, he will select the top four or five jumpers and begin the coaching process to help them reach even greater heights. In this manner, the coach has learned about the ability of the try-out jumpers based upon a standard scale (feet and inches). The coach then sets a standard (4 feet) for membership on the team, based upon his knowledge of what is necessary to score points at track meets (the competency standard).

How A CAT Exam Works

The above high-jump illustration can describe the way a CAT exam works. Every item within a live item pool has been calibrated to determine its level of difficulty. As each candidate takes an exam, the computer adaptive test must learn the ability level of the candidate.

The test typically starts with an item being administered that is slightly below the passing standard. The item may be from any subject area in the test plan:

• Airway, Respiration & Ventilation
• Cardiology & Resuscitation
• Trauma
• Medical/Obstetrics/Gynecology
• EMS Operations

After the candidate gets a short series of items correct, the computer will choose items of a higher ability, perhaps near entry-level competency. These items will also be taken from a variety of content areas of the test plan. If the candidate answers most of the questions in this series of items correctly, then the computer will choose new items that are at a higher ability level. Again, if the candidate answers many of these items correctly the computer will again present the candidate with items of an even higher ability level. Eventually, every candidate will reach his or her maximum ability level (and answer questions incorrectly). The computer then determines whether or not the individual is above the standard (entry-level competency) in these content areas, and the examination ends.

A 95% Confidence is Necessary to Pass or Fail a CAT Exam

The high achiever who is able to answer most of the questions correctly will find that the computer ends the exam. A candidate may worry that something is wrong because the exam was so short, when in reality, the computer was able to determine that the candidate jumped far higher than the standard level—or was well above the level of competency In a CAT exam.

The computer stops the exam when it is 95% confident that the individual candidate has reached the level of competency, is 95% confident the individual cannot reach the level of competency, or has reached the maximum allotted time. Thus, the length of a CAT exam is variable. Sometimes a candidate can demonstrate a level of competency in as few as 60 test items. Sometimes, after 60 questions, the candidate has shown to be close to entry-level competency but the computer has not determined within the 95% confidence requirement that the candidate is either above or below the entry-level competency standard. In cases when the computer is not 95% confident, the test continues to provide additional items. Each additional test item provides more information to determine whether or not a candidate meets the entry-level of competency. Regardless of the length of the test, items will still vary over the content domain (Airway/Oxygenation/Ventilation, Cardiology, Medical, Trauma, and Operations).

When (and if) the candidate reaches the maximum length of an examination, the ability estimate of that candidate will be most precise. Using the high jumper example, the computer will be able to determine those who jump 3 feet 11 inches from those who jump 4 feet 1 inch. Those who clear 4 feet more times than they miss 4 feet will pass. Those who jump 3 feet 11 inches but fail to clear 4 feet enough times will fail and are required to repeat the test. Some candidates won’t even be able to jump close to four feet. These candidates are below or well below the entry-level of competency. This too can be determined fairly quickly, and these candidates may have their examination ended quickly. When the examination is near 70 questions (for EMTs, or 90 for EMRs) and a candidate fails, he or she has demonstrated within 95% confidence that he or she cannot reach the entry-level of competency.

In a CAT exam, it is important that the candidate needs to answer every question to the best of their ability. The CAT exam provides the candidate with more than adequate opportunity to demonstrate their ability and is able to provide precision, efficiency, and confidence that a successful candidate meets the definition of entry-level competency and can be a Nationally Certified EMS provider.

Computer Based Linear Tests

Computer based linear tests are fixed length exams. The Advanced-EMT exam is currently a Computer Based Linear Test.

Unlike CAT exams, candidates who take a linear exam can skip questions, mark questions for review, and go back and change their answers provided time has not expired. There is no penalty for guessing. Any questions that are left blank are scored as incorrect.

---

Exam Results

Exam results are posted on the National Registry’s password-secure website through an individual’s login account usually within two business days following the completion of the examination provided you have met all other requirements of registration. Those candidates who pass the exam are sent National EMS Certification credentials by the National Registry. Candidates who successfully demonstrate entry-level competency do not receive specific details regarding their examination results, as it is not necessary.

Candidates who fail to meet entry-level competency are provided information regarding their testing experience. Studying examination items to prepare to do the job of an EMT is not helpful. Studying the tasks and the job of an EMT provides the best preparation. Candidates who memorize items in hopes of “getting them right,” the next time are wasting their time because masking items prevents them from seeing the same item again.

What is included on the exam? The "Test Plan"

National Registry examinations are developed to measure the important aspects of out-of-hospital care practice. Examination items are developed in relation to tasks identified in the practice analysis. The domain of practice that limits therapy addressed in an item is based upon National EMS Scope of Practice Model and the National Registry Practice Analysis.

EMS education programs are encouraged to review the current National Registry practice analysis when teaching courses and as a part of the final review of the abilities of students to properly deliver the tasks necessary for competent patient practice.

---

Based on the 2014 Practice Analysis, the current National EMS Certification Examinations cover five content areas:

• Airway, Respiration & Ventilation
• Cardiology & Resuscitation
• Trauma
• Medical/Obstetrics/Gynecology
• EMS Operations

All sections, except EMS Operations, have a content distribution of 85% adult and 15% pediatrics.
 

Content Area	EMR (90-110 items)	EMT (70-120 items)	Advanced EMT (135 items)	Paramedic (80-150 items)
Airway, Respiration & Ventilation	18%-22%	18%-22%	18%-22%	18%-22%
Cardiology & Resuscitation	20%-24%	20%-24%	21%-25%	22%-26%
Trauma	15%-19%	14%-18%	14%-18%	13%-17%
Medical/Obstetrics/Gyn	27%-31%	27%-31%	26%-30%	25%-29%
EMS Ops	11%-15%	10%-14%	11%-15%	10%-14%

National EMS Practice Analysis

The goal of licensure and certification is to assure the public that individuals who work in a particular profession have met certain standards and are qualified to engage in practice (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999). To meet this goal, the requirements for certification and licensure must be based on the ability to practice safely and effectively (Kane, 1982). The practice analysis is a critical component in the development of a legally defensible and psychometrically sound credentialing process.

The primary purpose of a practice analysis is to develop a clear and accurate picture of the current practice of a job or profession, in this case the provision of emergency medical care in the out-of-hospital environment. The results of the practice analysis are used throughout the entire National Registry of Emergency Medical Technicians (National Registry) examination development process, which helps to ensure a connection between the examination content and actual practice. The practice analysis helps to answer the questions, "What are the most important aspects of practice?" and "What constitutes safe and effective care?" It also enables the National Registry to develop examinations that reflect contemporary, real-life practice of out-of-hospital emergency medicine.

The National Registry conducted its first practice analysis in 1994 and at five year intervals. The most recent practice analysis was conducted in 2014. After the data is collected, an analysis is conducted that accounts for frequency and criticality. This weighted importance score is then combined for each of the five domains on the National Registry examinations: Airway, Respiration & Ventilation, Cardiology & Resuscitation; Medical & OB/Gyn, Trauma, and EMS Operations. After the weighted importance scores are calculated, the proportion represented by each area is used to set the blueprint for the next five years of National Registry examinations. A copy of the 2014 Practice Analysis is available for free, download it here.

Example Items

To help you understand what to expect from exam questions, we have provided examples of the types of questions entry level providers are expected to answer on the exam. These three questions range in difficulty but are all either at the passing standard or above the passing standard.

1. Your patient fell while skateboarding and has a painful, swollen, deformed lower arm. You are unable to palpate the radial pulse. You should immediately
   1. apply cold packs.
   2. align the arm with gentle traction.
   3. splint the arm in the position found.
   4. ask the patient to try moving the arm.
2. Your patient is a disoriented 86 year old male who has a history of terminal brain cancer. He fell out of bed and complains of severe right hip pain. His wife called 9-1-1 for assistance in putting him back to bed. She tells you that he has DNR orders and that she does not want him transported. You should initially
   1. explain why he needs to be transported.
   2. ask to see the DNR orders
   3. have her sign a refusal form.
   4. call for law enforcement back-up.
3. You are called to a scene where law enforcement officers have detained a man who they thought was drunk. They called you because he has a history of diabetes. You administer oral glucose and within a minute the patient becomes unresponsive. You should
   1. remain at the scene under law enforcement authority and request ALS back-up.
   2. leave the glucose in place, complete a primary survey and transport.
   3. open his airway with an oropharyngeal airway and ventilate him.
   4. suction his mouth and begin transport.

Where do the test questions (items) come from?

Individual examination items are developed by members of the EMS community serving on Item Writing Committees convened by the National Registry. Item Writing Committees typically have nine to ten national EMS experts as members (physicians, state regulators, educators and providers). They meet over several days to review, rewrite, and reconstruct drafted items. Consensus by the committee must be gained for each item. This ensures that every question is in direct reference to the tasks in the practice analysis; that the correct answer is the one and only correct answer; that each distracter option has some plausibility; and the answer can be found within commonly available EMS textbooks. Controversial questions are discarded and not placed within the pilot item pools. Items are also reviewed for the appropriate reading level and to ensure no bias exists related to race, gender or ethnicity.

Following completion of the item-writing phase, all items are pilot tested. Pilot items are administered to all candidates during the computer based cognitive exams. To the candidates, pilot items are indistinguishable from scored items; however, they do not count for or against the candidate. Extensive analysis of the performance of the pilot items is conducted with those functioning properly, under high stakes pilot testing. When the item analysis is complete, the items are determined to be functioning properly and are psychometrically sound; they are placed in “live” item pools.

The National Registry conducts differential statistical analysis of items in pre-test item pools and live item pools on an annual basis. Panels are then convened to review the items that show differential statistics and decisions are made regarding maintenance of any items within the pools.