Frequently Asked Questions
A N S W E R S
Q: What is progress testing?
A: Progress testing is an objective method for assessing the acquisition and retention of knowledge over time relative to a program's end-of-curriculum objectives.
- objective — uses multiple choice questions
- acquisition and retention — is concerned with how much you've retained of what you've learned since entering the program
- knowledge — measures your knowledge, not your clinical skills or your professional behaviour
- over time — involves multiple tests administered to you and your peers at routine intervals in the curriculum
- end-of-curriculum objectives — what the program expects you to know by the time you graduate or shortly thereafter
Expressed more simply, progress testing is longitudinal testing of knowledge acquisition.
Q: Is progress testing a new evaluation method?
A: No. Progress testing was pioneered in the 1970s by the medical schools at the University of Missouri (Kansas City), USA, and the University of Maastricht (Limberg), Netherlands. Since then, several dozen undergraduate medical schools and allied health sciences programs around the world have adopted progress testing in one form or another as a tool for assessing their students' learning profiles.
The particular method used by the IPPT (International Partnership for Progress Testing) was developed in the the early 1990s by the School of Medicine at McMaster University, Hamilton, Canada. Initially a pen and paper exercise using printed booklets and optical-mark response sheets, the test and the associated feedback for examinees have been administered entirely via an internet website since 2004.
Q: What is the basic goal of progress testing?
A: The basic goal is to help you, the learner, determine whether you are progressing adequately relative to both your peers and your program's objectives. Put another way, the goal is to determine whether you are..
- learning enough
- retaining enough
- doing so quickly enough
Q: Can you be more specific regarding me, the learner?
A: Of course. The aim is to help you assess your progress through the curriculum by providing you with formative feedback — information about your performance that highlights your strengths and weaknesses and, if warranted, will encourage you to modify your learning objectives and behaviours. Such feedback must..
- be informative
- about the extent of your knowledge base
- about your ability to self-assess your knowledge strengths and deficiencies
- sufficiently detailed to be instructive yet not misleading (results from too few items will be misleading)
- be timely and constructive
- reassure you if you are progressing adequately
- alert you if you are not
- alert you early enough to facilitate effective remediation
- not impact negatively on desirable learning behaviours, including..
- tutorial and classroom functioning
- peer, tutor, instructor and preceptor assessment
Q: What are the goals regarding my Program?
A: The goal is to help your educational program..
- identify and respond effectively to perceived deficits in some aspect of the curriculum, including content, methods and resources
- collect defensible evidence to support proposals for changes in the curriculum
That said, the first priority of progress testing is to help the the learner rather than the program.
Q: How does a progress test differ from the usual end-of-course examination?
A: A progress test differs from an end-of-course examination in several fundamental ways.
For the usual end-of-course examination..
- only those currently enrolled in the course sit the examination
the exam content..
- encompasses only the course content
- usually focuses on the course's core content; that is, the material, topics and concerns that the instructor(s) regard as most important for that course
- the instructors' aim is generally to discover if the student has learned and retained, for the moment at least, enough of the core content to be said to have "passed" the course
- pre- or post-test, the instructor will identify a score that constitutes the "passing grade" for the exam
- students must often attempt a specified minimum number of items (questions)
- most students attempt as many items as they can, including some for which the student guesses or invents an answer in the hope that something in what they provide may be rewarded
- whose results fall below the "passing grade" are classed as having "failed" the course
- often receive little or no feedback about their performance other than the bare statement of their total score
- are seldom provided with any perspective on their results, such as a breakdown of their scores by sub-divisions of the course content or historical data showing how well students in previous years performed on the exam
In contrast, for a typical progress test..
- all students enrolled in the program sit the same test (are presented with the same set of items), regardless of the student's level (academic year) in the curriculum
the test content..
- encompasses the whole domain of the Program's discipline
- ranges from what could be termed core knowledge through to what the Program expects its students to know by the time they graduate or what they are likely to know three to six months into their post-graduate training (eg, in a medical residency program)
students are instructed..
to attempt only items for which they have at least some knowledge about the subject or problem
As a result first-year students sitting their first progress test attempt far fewer items than students who are about to graduate, even though both groups may have the same percentage of correct responses among the items they attempted.
not to guess — doing so only confounds the student's capacity, post-test, to judge how well
she self-assessed her knowledge base
To discourage guessing, a statistical penalty is often applied to all examinees' scores.
- to attempt only items for which they have at least some knowledge about the subject or problem
- the test administrator does not define a "passing grade"; thus even those students having the lowest scores among their peer group are not classed as having "failed" the test
- students are provided with standardized versions of their scores (z-scores) to enable them to more effectively compare their results across multiple progress tests
students are given summary data to enable them to compare their performance to that of their peer group
and that of more junior or senior peer groups in the program (ie, students at a higher or lower academic year)
Students are encouraged explicitly to make such comparisons in thinking about their advancement through the curriculum.
- the program assesses the student's progress by comparing the student's results to those of the student's peer group on both the most recent test and any earlier administrations of the test post-test, the program advises students who seem not to be progressing adequately about how they might address their apparent learning deficiencies
Q: How is my performance on a single test assessed?
A: In the IPPT's version of progress testing..
performance is assessed by one measure — the number of correct responses on the whole test,
adjusted for guessing, and expressed as a percentage of the number of items on the test
If you have any incorrect responses, the adjustment for guessing yields a lower correct-response count; one-quarter of the number of incorrect responses is subtracted from the number of correct responses. Items that were not attempted are not considered incorrect.
performance is relative; that is, it is norm-based.
Your score is compared to the average score for your peers on the same test.
(Your peers are the students in your current cohort/class.)
- the comparison uses a standardized score (z-score), which expresses your score relative to the mean score for your peers
- if your score is relatively close to the mean score for your peers, then you and your program should conclude that your performance on that test is satisfactory. This is the case even if your score is below that mean.
if your score is well below your peer mean,
then you and your program should conclude that your performance on that test
is not satisfactory and that you may need to engage in some formal or informal remediation
Each educational program defines its own cut point (score) below which a score is said to be "unsatisfactory".All IPPT programs define one or more additional cut points above the threshold for "unsatisfactory". These additional boundaries enable the program to distinguish different degrees of "satisfactory".
Q: How is my "progress" assessed?
A: In the IPPT's version of progress testing..
- progress, or failure to progress satisfactorily, is demonstrated by comparing your performance across multiple tests, in sequence, from the first to the most recent, relative to your peer group's mean score on each test
- if your scores on all or most of the tests you've sat are relatively close to the mean scores for your peer group, then your program will normally conclude that you are progressing, that is, you are acquiring, retaining and applying your professional knowledge at a satisfactory rate. This will be the case even if your scores are consistently below those group means.
- if, however, your scores are well below the peer group mean on too many occasions, your program may conclude that you are not keeping up with your peers; that is, you are not progressing or not progressing at a satisfactory rate.
- each educational program defines its own criteria for identifying failure to progress or failure to progress at a satisfactory rate.
Q: What is the "PPI"?
A: "PPI" is an alias for "Personal Progress Index", which is the formal name used since 1992 by McMaster University's health sciences programs for their implementation of progress testing.
The IPPT website and online testing application both refer to your program's progress test as "the PPI".
Your program may use another name in its communications with you for the PPI-type tests it administers via its IPPT-based website.
Q: Is the PPI a proven evaluation tool?
A: Yes. Several reviews of past PPIs have established a number of stable and reliable patterns in the performance measures assessed by the McMaster PPI. (The primary measure is the whole-test PAC (Percentage Adjusted Correct) or the percentage correct after adjusting for guessing. Other whole-test measures are the percentage attempted and the percentage adjusted correct among items attempted.)
The principal findings include..
- patterns are relatively stable across tests (Ts) and across cohorts
- means at any given point in the curriculum are relatively consistent across tests and across cohorts
- examinee performance is relatively consistent across tests and classes*
* Computed from results on 17 tests administered to six classes by the
- overall test reliability: 0.9 (T1 to T8), 0.7 (2 tests)
- test-retest correlation: 0.6 – 0.7 (T3:T4 to T7:T8)
McMaster Undergraduate Medical Program, Oct 2005 to Feb 2011.
Q: When will I be tested?
A: That depends on your program. Ask your program's PPI Test Manager.
Most IPPT-partner programs administer the first PPI of each academic year in the first month of the year. Some programs administer three PPIs during the academic year. Other programs administer only two.
Q: Do I have to sit the PPI?
A: That's up to your program. Most IPPT-partner programs require all students in all cohorts/classes to sit all administrations of their PPI.
Q: Should I study for my next PPI?
A: No. Truly. Don't study for any upcoming progress test. You'll only be wasting your time, confounding the diagnostic value of your results, and very likely raising your stress levels.
- The 180 or so items on the test are selected randomly from a large item bank whose content encompasses the entire domain of medicine. What's the likelihood that you will be lucky enough to focus on the material covered by those items?
- Remember that the chief goal of progress testing is to help you discover the extent of your stable, persistent knowledge. It's not to see how much you can cram into your head a day or two before the test.
- Most of what's learned in an intensive pre-test 'cram' session will be forgotten within a few weeks, so there's little lasting benefit from studying for the test.
If you've been keeping up with your classroom, tutorial and other work,
you are as prepared as you need or should be.
Studying for a PPI will distort your ability to self-assess the strengths and weaknesses in your current knowledge base.Moreover, time and energy allocated to studying for the test will likely impact negatively on your performance in other learning and assessment contexts, including tutorials, lectures, and clinical settings.
Q: What does a PPI test consist of?
A: Each PPI has these basic characteristics..
- Each test consists of a number of five-option, A-type (best single response) multiple-choice items.
Each educational program decides how many items will appear on its tests.
Most partners draw 180 items, assuming the average examinee will be able to read and respond to an item of average difficulty in a minute or less.
Q: What do I do to take the PPI?
A: The procedure is quite simple.
- You will go to a location on a date and at a time provided to you by your program's Test Manager or Administrator.
- After logging on to a computer in that location, you will log in to your Program's online PPI testing site, using a url, user name and initial password provided to you.
- Once the Test Manager or Site Invigilator has started the test, you will be able to access the test instructions, followed by the first of several pages displaying items to be answered.
- Each test-item page except possibly the final page displays the same number of items. (Twenty items in the case of a 180-item test.)
Each test-item page provides links to additional resources, including
- a timer that reports the number of minutes to the scheduled end of the test or the number of minutes since you started the test
- a calculator
- a link to tables of normal clinical values
- a table displaying the status of each item on the page, indicating those you have answered, those you have not answered, and those you have answered but want to return to before submitting the page
After you are satisfied with your response to each item on the current
test-item page, you can submit your responses by clicking
a "Submit" button.
Do not click this button until you are certain that you are ready to leave the page.
You can not return to any test-item page after submitting it.
- The web site's server will return the next test-items page, if any. Otherwise, it will return a page inviting you to view a simple summary of your performance on the test.
- You will log out of the testing site.
- A few days later, after everyone who was required to sit the test has done so, your Program's Test Manager or Administrator will enable release of the test results and inform all examinees that enhanced reports are now available.
You can then log on to the testing site from any location,
go to your My Account (My Profile) page, view and print..
- an expanded summary of your performance on the latest test, one that now includes summary statistics for each peer group (class/cohort) that sat the test.
- a report displaying your primary whole-test score (your PAC, or percentage adjusted correct) on each test you've sat thus far.
Q: What are some good test-taking strategies for the PPI?
A: Here's the best advice we can give to any student about to sit a PPI-type progress test:
Don't study for the test.
(See the question above for some reasons why you shouldn't.)
Attempt only those items for which you have some knowledge and are reasonably confident you know or can deduce the correct response.
- Each student will have her own comfort zone regarding how confident she must be about an item before deciding to attempt it.
- Some students are comfortable if they can reason their way through to eliminating all but two options; they then select the most likely option, based again on their limited knowledge, doing so even if they are none too confident in their final choice. Other students will be more or less demanding.
- We suggest that absolute certainty is not a good requirement for deciding whether to attempt an item.
Don't guess. It's counter-productive in at least two ways.
- First, there's a statistical penalty for wrong answers (# adjusted correct = #correct – 25% of #wrong).
Second, guessing reduces the diagnostic value of the feedback you will receive.
Remember that a major goal of progress testing is to help you test your ability to assess the strengths and weakenesses in your knowledge base. What you attempt is a measure of what you think you know. Don't confound that measure by including guessed items.
Q: What feedback will I receive and how do I obtain it?
A: Feedback is provided at two points in the testing process and is also available at any time between test administrations.
- During a test — after you submit the last test-items page, the web site will return a page inviting you to view a simple summary of your performance on the test. This summary displays no peer group results.
A few days later — after the last student scheduled to sit the test has done so,
your Program's Test Manager or Administrator will enable access to peer
results. Now, if you log in to your My Account (My Profile) page
on the PPI test site, you can view..
- an enhanced Performance Report that includes performance data for each peer group (class/cohort) that sat the test. This report is explained on our "Understanding your Performance Report" page.
- the report that displays your whole-test PAC score for every test you've completed thus far. This report, which is usually identified as the Zone Report, is explained on our "Understanding your Zone Report" page.
- Thereafter, you can return to your Profile page at any time, from any internet-enabled location, and access your Performance Report for any test you've sat or your multi-test Zone Report.
- Performance and Zone reports can be saved to your computer and/or printed whenever you view them.
Q: How will my Program respond to my performance on successive PPIs?
A: That varies. Each IPPT-partner Program defines its own criteria and procedures for dealing with students who may not be progressing or not progressing at a satisfactory rate. Contact your Test Manager or Program Administrator for the details.