Conference Close-up: Perfecting the Test Through Question Analysis
Posted by Joan Phaup
Neelov Kar, Project Management Program Owner for Dell Services (previously Perot Systems) is getting ready to attend the Questionmark Users Conference in Miami this month. He will be delivering a case study about how he and his team have used statistical analysis to improve their test questions. I spent some time talking with Neelov the other day and wanted to share what I learned from him.
Q: Tell me a little about your company.
A: We are a one-stop shop for IT Services and have people working all over the world, in 183 countries.
Q: What does your work entail?
A: I’m the project management program owner, so I am in charge of all the project management courses we offer. I help identify which courses are appropriate for people to take, based on training need analysis, and I work with our project management steering committee to work out what courses we need to develop. Then we prioritize the requirements, design and develop the courses, pilot them and finally implement them as a regular course. As a Learning and Development department we also look after leadership courses and go through a similar process for those. I moved into this role about a year ago. Prior to that I was leading the evaluation team, and it was during my time on that team that we began using Questionmark.
Q: How you do you use online assessments?
A: We use Questionmark Perception for Level 2 assessment of our project management and leadership courses. We started with a hosted version of Questionmark Perception and it was I who actually internalized the tool. We offer leadership courses and project management courses internally within the organization across all geographies. Some of the project management courses already had tests, so we converted those to Questionmark. We started designing the end-of-course assessments for our newly introduced leadership and project management courses once we started using Perception.
Q: What you will be talking about during your conference presentation?
A: Last year we introduced a new course named P3MM Fundamentals, and because it was a new course we had to pilot the course with some of our senior members. In the pilot we asked the students to take the end-of-course test, and we found that many people had trouble passing the test. So we analyzed the results and refined the questions based on the responses. Analysis of results within Perception — particularly the Assessment Overview Report, Question Statistics Report, Test Analysis Report and Item Analysis Report — helped us in identifying the bad questions. We also saw that there were things we could do to improve the instruction within the course in order to better prepare people for the test. Using the Questionmark reports, we really perfected the test. This course has been going for over a year now, and it’s pretty stable. Now, every time we launch a course we do a pilot, administer the test and then use the Questionmark tools to analyze the questions to find out if we are doing justice to the people who are taking the test.
Q: What are you looking forward to at the conference?
A: I want to find out what Perception version 5 offers and how we can use it for our benefit. Also, I saw that there are quite a few good papers to be presented, so I’m looking forward to attending those. And I want to get involved in the discussion about the future of SCORM.
Neelov’s is just one of 11 case studies to be presented at the conference, which will also include technical training, best practice presentations, peer discussions and more. Online registration for the conference ends on Tuesday, March 9th, so if you would like to attend, be sure to sign up soon!