How many items are needed for each topic in an assessment? How PwC decides

Posted by John Kleeman

I really enjoyed last week’s Questionmark Users Conference in Los Angeles, where I learned a great deal from Questionmark users. One strong session was on best practice in diagnostic assessments, by Sean Farrell and Lenka Hennessy from PwC (PricewaterhouseCoopers).

PwC prioritize the development of their people — they’ve been awarded #1 in Training Magazine’s top 125 for the past 3 years — and part of this is their use of diagnostic assessments. They use diagnostic assessments for many purposes but one is to allow a test-out. Such diagnostic assessments cover critical knowledge and skills covered by training courses. If people pass the assessment, they can avoid unnecessary training and not attend the course. They justify the assessments by the time saved from training not needed – being smart accountants using billable time saved!

imagePwC use a five-stage model for diagnostic assessments: Assess, Design, Develop, Implement and Evaluate as shown in the graph on the right.

The Design phase includes blueprinting, starting from learning objectives. Other customers I speak to often ask how many questions or items they should include on each topic in an assessment, and I thought PwC have a great approach for this. They rate all their learning objectives by Criticality and Domain size, as follows:

Criticality
1 = Slightly important but needed only once in a while
2 = Important but not used on every job
3 = Very important, but not used on every job
4 = Critical and used on every job

Domain size
1 = Small (less than 30 minutes to train)
2 = Medium (30-59 minutes to train)
3 = Large (60-90 minutes to train)

The number of items they use for each learning objective is the Criticality multiplied by the Domain size. So for instance if a learning objective is Criticality 3 (very important but not used on every job) and Domain size 2 (medium), they will include 6 items on this objective in the assessment. Or if the learning objective is Criticality 1 and Domain size 1, they’d only have a single item.

I was very impressed by the professionalism of PwC and our other users at the conference. This seems a very useful way of deciding how many items to include in an assessment, and I hope passing on their insight is useful for you.

3 Responses to “How many items are needed for each topic in an assessment? How PwC decides”

  1. Silvester Draaijer says:

    John,

    Nice post. Did PwC also bring to the attention whether their test are norm-referenced or criterion-referenced? I take it from their approach that they deploy criterion-referenced tests. Maybe a nice link could also be taken up here: http://en.wikipedia.org/wiki/Criterion-referenced_test

    Kind regards,

    Silvester

  2. John Kleeman says:

    Silvester

    Yes, I confirm that they use criterion-referenced tests, as is common in the coporate world. Thanks for the reference.

    John

  3. […] Verdeel de te behalen scorepunten zodanig dat belangrijke onderwerpen/vaardigheden meer vragen krijgen (MC-deel) of meer punten (subvragen) krijgen (open deel). Van belangrijker onderwerpen wil je namelijk meer informatie verzamelen om de uitslag op te baseren. Een aardig artikel hierover is bijgaande: http://blog.questionmark.com/how-many-items-are-needed-for-each-topic-in-an-assessment-how-pwc-decid…. […]

Leave a Reply