Development Process for New Physical Therapy, Occupational Therapy, and CNA Assessments

The content validation process takes the Uniform Guidelines on Employee Selection Procedures (1978) and applies them to the development of these assessments.

The process for each assessment specialty was performed in two phases.

Phase One

The first phase involved an in depth Job Analysis that was performed using 7 subject matters experts (SME’s) from around the country for both the Physical Therapy, Occupational Therapy, and Certified Nursing Assistant assessments. The subject matter experts, through a facilitated workshop, identified the key job duties/tasks and specific knowledge, skills, and abilities that PT and OT clinicians and CNAs need to possess on the first day of the job. The panel of subject matter experts provided a series of ratings and linkages.

Phase Two

Once the Job Analysis phase was completed, the subject matter experts developed a test plan and began the item-writing phase for each assessment. This phase involved specific criteria for item writing that conforms to the recommendations from the Dept of Labor and the Uniform Guidelines on Employee Selection Procedures (1978).

During both phases, a significant amount of statistical data was collected and analyzed, such as inter-rater reliabilities and individual item Angoff ratings, yielding a final recommendation as to which items were to be included on each assessment and what the initial unmodified Angoff Cutoff Score should be for each assessment. The unmodified Angoff Cutoff score was generated by having each subject matter expert provide their rating regarding the percentage of minimally qualified applicants they believe will answer the assessment item correctly.

After detecting and removing outlier Job Experts from the data set, the unmodified Angoff Cutoff Score became the average of the Job Expert panel’s ratings for each item on the assessment. Each item’s averaged percentage rating received equal weight in calculating the unmodified Angoff.

Form A and Form B – Parallel Form Development

Parallel forms are constructed so that each form can be used independent of one other and be considered equivalent measures. Parallel forms measure comparable content and produce scores that can be interpreted the same way. The first exam form is administered to the applicant for testing. Should retesting be necessary, the applicant would be given the second form. In order to achieve equivalent reliability for both forms, Amistaff builds parallel forms based upon several components of the test item surveys that have been completed by our panel of subject matter experts (SME’s) for that particular clinical specialty.

All questions are reviewed by the SME’s to decide whether test items measure factors including, but not limited to, being correct, fair to all groups of people, measures a knowledge, skill or ability necessary on the first day of the job, necessary to be memorized, etc. After reviewing all aggregate data, test items that are “rejected” in such areas as those mentioned above may be removed from the test item bank. For example, if the SME’s do not agree that the correct answer is tagged for an item, that particular item may be removed from the total group from which parallel forms are constructed. See sample below on how test items are reviewed for these specific areas.

After removing necessary items from the item bank due to rejection in the above categories, the remaining items are collected and the average Angoff rating for each item is evaluated and sorted in descending order. All questions with an average Angoff rating of 90% or above were removed from the item bank, due to the high probability of those items having a high percentage of applicants selecting the correct answer, thus not contributing significantly to the validity and overall reliability of the exam(s). Remaining items were then equally distributed into Form A and Form B with similar average Angoff ratings, so as to create equality in both Form A and Form B. See below for an example of how items that survived the first round of scrutiny from the SME’s were removed due to a high Angoff rating. The average Angoff rating for Form A and Form B was then calculated as items were placed into each form, to assure that the unmodified Angoff score was as close to one another as possible.

For a better understanding of the Angoff Cutoff Scores, see the blog post titled New Assessment Data Allows Employers to Make Informed Scoring Decisions.

All future exam content will undergo this same development procedure so that NurseTesting can continue to provide the best prescreening assessment tools for our clients!