Posted by Joan Phaup

Steve Lay

Daniel R. Rehak, a technical advisor to the Advanced Distributed Learning Initiative (ADL), is responsible for technical and strategic advice primarily about the direction and refinement of ADL’s activities in the development of SCORM and content repositories and registries. As part of its plan to release an enhanced, harmonized version of its SCORM interoperability standards in 2011, ADL is actively seeking the views of SCORM and AICC community members. Dan is coming to the Questionmark Users Conference, takinig place in Miami March 14 – 17,  to hear what customers hope to see in an updated version of SCORM. He and Questionmark Integration Team Lead Steve Lay will co-facilitate a discussion on Shaping the Future of ADL SCORM:  What’s On Your Wish List?

I recently asked Dan about his ADL work and his desire to sit down with Questionmark users to discuss the future of SCORM:

Q: How would you define SCORM?

Daniel Rehak

A: SCORM is a model — a collection of different interoperability standards that for the most part were originally created to work on their own. SCORM explains how to make them work together with the objective of taking learning activities, managed by a learning management system, and making them portable, interoperable and reusable across different vendors and technology platforms. In addition to being used by many commercial organizations and the military, it’s used for K12 education in the UK and Korea. It’s used widely all over the world, notably in higher education and industry.

Q: Could you tell me about your role with ADL SCORM?

A: I’m involved in planning the ADL SCORM update: looking at the form the update will take, who we should talk to, the timeline, the tasks we need to do,  the technical direction, and then communicating those out to various people: vendors, users, and their organizations both here in the US and internationally. I’m also helping ADL upgrading its registry federation model for learning content.

Q: Can you share some details about the planned update to SCORM?

A:  SCORM has not been through a major update since 2004, although we made some minor, incremental changes along the way.  By talking with vendors and user communities, we are finding divergent views about what’s important and what’s lacking — those kinds of things. We know that technology has evolved on the Web.  We also know we have legacies out there: we have people procuring SCORM-based delivery systems today with the expectation that they’re going to be around in a number of years and that they are still going to work. So the objective is to produce an update that is grounded in the reality of what people want to use and what they actually have in place. We have to balance legacy support requirements with new capabilities and requirements. Our goal is to produce an updated, revised and harmonized version of SCORM by late 2011. We are talking to users, talking to adopters and to vendors to find out what things they are planning to put into their products in the short term – say the next 18 months or so. We may not be able to do everything we want to do in that  timeframe, but  our assumption is that things that don’t hit this cycle will get into the next cycle.

Q: What issues are you finding as you talk with people about the future of SCORM?

A: A number of things keep coming up: Web services, different content exchange models, content mark-up languages, assessments – those are the kinds of things that are on the list. What we are trying to do is develop a flexible model that indicates what we feel is important and that vendors can get behind. We also want organizations to be able to add features of their own in harmony with what we are doing. We know we won’t be able to do everything ourselves; we are relying on the community to bring their resources to bear on this and help us collectively move things forward.  We call it harmonization because we want to bring people together and move forward together.

Q: What’s prompted you to attend the Questionmark Users Conference?

A: We’re reaching out to a number of different vendors to find out what their users are interested in. We’ve been in discussions with Questionmark for awhile and this came up as an opportunity to talk with this user segment. Assessments standards are not currently in SCORM, but we know assessments and tracking are things people have said are very important. You have one of the best user communities there to talk to, so we want to talk with them and find out what they see as being important in SCORM as we go forward.

Q: Who should attend this session?

A: Anyone from the SCORM community.  We’ve got different communities that have different levels of adoption. Also, anyone who uses AICC, which has some elements in common with SCORM. They have a significant overlap in what they do and how they do it. We have been in direct communication with AICC about the harmonization and would welcome ideas from their users. Also, from the standpoint of specifications and standards, we’re interested in hearing from anyone with opinions about IMS QTI.

Q: What kind of input are you looking for during your session about the future of SCORM?

A: We want to know what people are currently doing; what problems they’re seeing; what they think in general about assessment and tracking. We’d like to learn more about where they do and don’t use SCORM and why. We’ll also ask what we can change in SCORM to make their lives better.

Come to the conference and let your voice be heard! You can register online or email conference@questionmark.com for further information.

Related posts:

1. Conference Close-up: Technical Standards and Questionmark’s Open Assessment Platform
2. Assessment Standards 101: SCORM
3. Conference Close-up: Perfecting the Test Through Question Analysis

Posted by Greg Pope

In my last post I discussed specific aspects of construct validity. I’m capping off this series with a discussion of modern views and thinking on validity.

Dr. Bruno D. Zumbo

Recently my former graduate supervisor, Dr. Bruno D. Zumbo at the University of British Columbia, wrote a fascinating chapter in the new book, The Concept of Validity: Revisions, New Directions and Applications, edited by Dr. Robert W. Lissitz. Bruno’s chapter, “Validity as Contextualized and Pragmatic Explanation, and its Implications for Validation Practice,” provides a great modern perspective on validity.

The chapter has two aims: to provide an overview of what Bruno considers to be the concept of validity, and to discuss the implications for the process of validation.

Something I really liked about the chapter was its focus on why we conduct psychometric analyses digging into how our assessments perform. As Bruno discusses, the real purpose of all the psychometric analysis we do is to support or provide evidence for the claims that we make about the validity of the assessment measures we gather. For example, the reason we would do a Differential Functioning Analysis (DIF), in which we ensure that test questions are not biased against/towards a certain group, is not only to protect test developers against lawsuits but also to weed out invalidity in order to help us set where the inferential limits of assessment results are.

Bruno drives home the point that examining validity is an ongoing process of validation. One doesn’t just do a validity study or two and then be done: validation is an ongoing process in which multilevel construct validation occurs and procedures are tied in to program evaluation and assessment quality processes.

I would highly recommend that people interested in diving more into the theoretical and practical details of validity check out this book, which includes chapters from many highly respected psychometrics and testing industry experts.

I hope that this series on validity has been useful and interesting! Stay tuned for more psychometric tidbits in upcoming posts.

————–

Editor’s Note: Greg will be doing a presentation at the Questionmark Users Conference on Conducting Validity Studies within Your Organization. The conference will take place in Miami March 14 – 17. Learn more at www.questionmark.com/go/conference

Related posts:

1. Understanding Assessment Validity: Construct Validity
2. Understanding Assessment Validity: An Introduction
3. Understanding Assessment Validity: Content Validity

I’ve been having a great time talking to presenters at the Questionmark 2010 Users Conference – customers, our keynote speaker and Questionmark staff. I wanted to find out from Howard Eisenberg about the Best Practices presentation he will deliver at the conference on Effectively Measuring Knowledge, Skill and Ability with Well-crafted Assessments

Q: Could you explain your role at Questionmark?

A: I manage Training and Consulting, so I work with our customers to get the most of their assessments and their use of Questionmark Perception. For some that might mean training on how to use the software effectively. For others it might mean providing solutions that allow them to use the software within the context of their current business processes, such as synchronizing the data between the organization’s central user directory and Perception. In some cases we might need to create reports to supplement those that come with the product or do some other custom development. Sometimes we go on site, install the Perception software and set it up within the customers’ LMS and do any troubleshooting right on the spot. Whatever we do, our goal is to ensure customers’ speed to success, getting them operational faster.

Q: What will you be talking about during your Best Practice session in Miami?

A:  Over the years I’ve given presentations on Creating Assessments That Get Results, where I cover the dos and don’ts of writing test items. A question that always comes up during those talks is how to write test content that goes beyond testing information recall…content that tests a person’s ability to perform a task. There are limitations to using software like Perception to do that: certain things simply require that a person perform a task and have someone observe them, so that all the scoring and evaluation is done by an observer or rater. But there are a lot of possibilities for creating computer-scored items that can measure skill and ability rather than just recall of information. This session is designed to give people tools to take their tests to that level. First we need a framework for categorizing knowledge, skills and abilities: what makes a skill a skill and an ability an ability. We’ll help people classify their learning objectives along those lines and look at specific types of questions that can be used to measure skill and ability. The questions that provide this kind of measurement expand upon the question types that are supported in Questionmark Perception—selected response types as well as constructed responses.   We’ll use several real-world examples to illustrate how questions of this nature go beyond recall of knowledge and go to skill and ability.

Q: What are you looking forward to at the conference?

A: I am really looking forward to meeting customers and in some cases reconnecting with customers I’ve gotten to know over the years. That’s really a highlight for me…reconnecting with our great customers. I am consistently amazed and impressed about how passionate our customers are about what they do with our software and how smart they are in using it. Every year, after talking with a customer or sitting in on a case study, I come away thinking, Wow! That was really clever! So I’m looking forward to hearing those kinds of stories again this year.

The conference program is nearly finalized and includes case studies, tech training and best practice sessions for every experience level.  Check it out and plan to join us March 14 – 17 in Miami!

Related posts:

1. Questionmark Conference Close-up: Making Assessments Accessible
2. Conference Close-up: Technical Standards and Questionmark’s Open Assessment Platform
3. Conference Close-up: Using Flash and Captivate Questions with Questionmark

Questionmark Analytics and Psychometrics Manager Greg Pope, like many other staff members, is busy preparing for the Questionmark Users Conference in Miami March 14-17.  Here’s a little background about Greg and a quick round-up of his activities at the conference.

Q: What’s your role at  Questionmark?
A: I have several roles at Questionmark. I am the product owner for reporting, so I’m always talking to customers to find out their requirements for reporting and plan the best ways to add new reports and reporting features into our software. I am also the in-house psychometrics expert, making sure the software we develop conforms to best practices and high standards. Finally, I do a lot of writing and presenting about psychometrics and other issues in the area of assessment.

Q: What will you be presenting at the Users Conference?
A: I have two Best Practice sessions: one on item and test analysis and one on conducting validity studies. I’m also helping out with a Peer Discussion about high-stakes testing.

Q: What would you say are the things most people want to learn about item and test analysis and analytics?
A: People want to know how to use the tools available to them in our software to make the best assessments possible and to make sure they conform to best practices. For example, they want to know what specific things to look for in the Item Analysis Report to find out how their questions are performing. They want to find out how to make their questions better and to make sure their assessments are measuring what they need to measure.

Q: Could you give some details about your session on conducting validity studies?
A:  People have an interest in what validity means and what they can do to evaluate the validity of their assessment program. You don’t have to hire out a team of Ph.Ds to do validity studies. With a solid knowledge of the concepts, and using tools like Excel, organizations can investigate validity in their contexts. I’ll share the theoretical background on validity and provide applied examples of conducting validity studies to help organizations conduct their own studies.

Q: Tell me more about the Peer Discussion, too!
A: We’ll cover some pertinent topics in high stakes testing and what the current thinking is about them. I’ll spend a little time sharing some best practices, but most of the session will be Perception users talking about what challenges they’ve encountered and how they’ve addressed them. Perhaps we will be able, as a group, to come up with some good approaches for particular situations.

Q: What are you look forward to most at the conference?
A:  I like having the opportunity to talk with customers at the Best Practice sessions and in Product Central to find out what issues people are encountering – not just the features they’d like in the software but what issues they’re encountering in the assessment industry as a whole…what kinds of tools they need and what kinds of knowledge they need in order to achieve their business goals.

Earlybird conference registration ends tomorrow, January 22nd. Learn more about the conference and register at www.questionmark.com/go/conference.

And just for fun, check out this brief video we put together after last year’s conference!

Related posts:

1. Conference Close-up: Technical Standards and Questionmark’s Open Assessment Platform
2. Conference Close-up: Assessments That Measure Knowledge, Skill & Ability
3. Conference Close-up: Perfecting the Test Through Question Analysis

Participants in the annual Questionmark Users Conference bring a lot of enthusiasm about using innovative question types in their assessments. A number of our customers have extensive experience with this and like to share their expertise at the conference. I spoke the other day with Doug Peterson from Verizon Communications and asked him about the case study he will share at the conference about using Flash and Captivate questions  within Questionmark Perception.

Here’s a quick wrap-up of our conversation:

Q:  What’s your role at Verizon  Communications?
A:  I have two roles: I develop, maintain and deliver training — mainly  now on internet technologies — and I’m responsible for a series of online automated tests for our help center training program. This is a pass/fail curriculum and very high stakes because these tests can affect people’s job status. So we need to be absolutely sure that the tests are well written and well maintained. These used to be written tests that were graded by an instructor. We turned to Questionmark for an objective, unbiased, online, airtight testing system and I oversee that.

Q: How are you using Questionmark Perception?
A: We have a couple of tests for each of the three modules in the training curriculum. We use Questionmark for end-of-lesson reviews as well as the higher stakes tests  that determine whether a person has passed or failed a module. We use scenarios that trainees might encounter in working with a customer. There might be 6 to 8 scenarios in each test and 10 or 12 questions about each scenario. The trainees take these tests right in the classroom, on their classroom computers.  We create individual QM accounts for each student and schedule the tests directly for those accounts.  We schedule them for a specific day and time window.  No one can see the tests except for the students, and they can only access them during the testing window. We had subject matter experts tell us what we needed to cover in the scenarios and what questions we needed to ask about them. They explained what would be a reasonable way to present a question or simulation to test a particular skill. Once we’d created all the scenarios and written all the questions we did an in-depth validation.

Q: What will you be sharing during your case study presentation at the Users Conference?
A:  Our call center agents have to use several applications when they get a call from a customer. They’ll have to look up a trouble ticket, get information about the customer and so forth. We need to make sure they knew how to use those applications, so we have created Perception questions using Captivate and Flash files with ActionScript that present the application to the student. Then the student needs to work through the application to demonstrate their proficiency with it. We’ve worked out a way to create a highly interactive, very realistic simulation in Flash that captures each student’s actions in using a particular application. It really tracks step by step. Being able to take the individual things from the Flash scenarios makes it so that when we run reports after the test we can easily see if a lot of people are is missing something like clicking on a particular button. Then the instructor can go back and make sure the students understand what they are supposed to. We went through a complex process to figure all this out, but it’s given us the ability to create a highly interactive, very realistic simulation in Flash with action script ActionScript coding and all kinds of logic and still pass back individual point values for different tasks. I’m very proud of the tests we have created and the work we have done. We have some fabulous questions in there that allows the students to show that they really understand applications and know how to do something from start to finish. We learned many tips and tricks along the way and I will be sharing those with the people at my session.

Q: What are you looking forward to at this year’s conference?
A: I really enjoyed the sessions on item analysis and test validity at the 2009 conference, and I am looking forward to learning even more about those subjects this year. And anything about new functionality in Perception Version 5 will be on my list too.

You can attend Doug’s presentations and many others at the conference in Miami March 14 – 17. Early-bird registration ends January 22nd, so sign up soon!

Related posts:

1. Conference Close-up: Assessments That Measure Knowledge, Skill & Ability
2. Conference Close-up: Questionmark Live at Shenandoah University
3. Conference Close-up: Perfecting the Test Through Question Analysis

The School of Justice at Miami Dade College in Florida plays an important role in helping people prepare for careers in public safety. Not only does the school offer degree programs, it is also home to the state’s basic abilities testing program for prospective police, correctional and probation officers.

The Florida Basic Abilities Test (F-BAT), which measures basic abilities of recruits planning to enter training programs, needs to be kept current and relevant. The College reviews the test annually and freshens the questions, drawing on the expertise of subject matter experts (SMEs) from all over the state.

The introduction  this year of Questionmark Live has given the college a new, more efficient  way of working with SMEs: During a recently item writing workshop held at the College, SMEs were given a quick tutorial on the use of Questionmark Live and were soon creating questions using its simple browser-based authoring tools. Their efforts were so successful  that the college plans to run future workshops at a distance, doing away with the need for SMEs to meet together in order to help create test content.

You can learn more about the college’s role here and find out more about the item writing sessions by listening to the following podcast with Lebsica Gonzalez, F-BAT project manager at Miami Dade College.

Related posts:

1. Conference Close-up: Bring-Your-Own-Laptop Item Writing Workshop
2. Podcast: Using Questionmark Perception for Placement Tests
3. How it Works: Writing a Multiple Choice Question in Questionmark Live

Posted by Greg Pope

In my last post I discussed content validity. In this post I will talk about construct validity. Construct validity refers to whether/how well an assessment, or topics within an assessment, measure the educational/psychological constructs that the assessment was designed to measure. For example, if the construct to be measured is “sales knowledge and skills,” then the assessment designed to measure this construct should show evidence of actually measuring this “sales knowledge and skills” construct.

It will come as no surprise that measuring psychological constructs is a complicated thing to do. Human psychological constructs such as “depression,” “extroversion” or “sales knowledge and skills” are not as straightforward to measure as more tangible physical “constructs” such as temperature, length, or distance. However, luckily there are approaches which allow us to determine how well our assessments accomplish the measurement of these complex psychological constructs.

Construct validity is composed of a few areas with convergent and discriminant validity being the core:

In my next post I will drill down more into some of these areas of construct validity.

Related posts:

1. Understanding Assessment Validity: Content Validity
2. Understanding Validity: More on Construct Validity
3. Understanding Assessment Validity: Criterion Validity

Posted by Greg Pope

In my last post I discussed criterion validity and showed how an organization can go about doing a simple criterion-related validity study with little more than Excel and a smile. In this post I will talk about content validity, what it is and how one can undertake a content-related validity study.

Content validity deals with whether the assessment content and composition are appropriate, given what is being measured. For example, does the test content reflect the knowledge/skills required to do a job or demonstrate that one grasps the course content sufficiently? In the example I discussed in the last post regarding the sales course exam, one would want to ensure that the questions on the exam cover the course content area of focus appropriately, in appropriate ratios. For example, if 40% of the four-day sales course deals with product demo techniques then we would want about 40% of the questions on the exam to measure knowledge/skills in the area of demo skills.

I like to think of content validity in two slices. The first slice of the content validity pie is addressed when an assessment is first being developed: content validity should be one of the primary considerations in assembling the assessment. Developing a “test blueprint” that outlines the relative weightings of content covered in a course and how that maps onto the number of questions in an assessment is a great way to help ensure content validity from the start. Questions are of course classified when they are being authored as fitting into the specific topics and subtopics. Before an assessment is put into production to be administered to actual participants, an independent group of subject matter experts should review the assessment and compare the questions included on the assessment against a blueprint. An example of a test blueprint is provided below for the sales course exam, which has 20 questions in total.

The second slice of content validity is addressed after an assessment has been created. There are a number of methods available in the academic literature outlining how to conduct a content validity study. One way, developed by Lawshe in the mid 1970s, is to get a panel of subject matter experts to rate each question on an assessment in terms of whether the knowledge or skills measured by each question is “essential,” “useful, but not essential,” or “not necessary” to the performance of what is being measured (i.e., the construct). The more SMEs who agree that items are essential, the higher the content validity. Lawshe also developed a funky formula called the “content validity ratio” (CVR) that can be calculated for each question. The average of the CVR across all questions on the assessment can be taken as a measure of the overall content validity of the assessment.

You can use Questionmark Perception to easily conduct a CVR study by taking an image of each question on an assessment (e.g., sales course exam) and creating a survey question for each assessment question to be reviewed by the SME panel, similar to the example below.

You can then use the Questionmark Survey Report or other Questionmark reports to review and present the content validity results.

So how does “face validity” relate to content validity? Well, face validity is more about the subjective perception of what the assessment is trying to measure than about conducting validity studies. For example, if our sales people sat down after the four-day sales course to take the sales course exam and all the questions on the exam were asking about things that didn’t seem related to the information they just learned on the course (e.g., what kind of car they would like to drive or how far they can hit a golf ball), the sales people would not feel that the exam was very “face valid” as it doesn’t appear to measure what it is supposed to measure. Face validity, therefore, has to do with whether an assessment looks valid or feels valid to the participant. However, face validity is somewhat important:  if participants or instructors don’t buy in to the assessment being administered, they may not take it seriously,  they may complain about and appeal their results more often, and so on.

In my next post I will turn the dial up to 11 and discuss the ins and outs of construct validity.

Related posts:

1. Understanding Assessment Validity: Criterion Validity
2. Understanding Assessment Validity: Construct Validity
3. Understanding Assessment Validity: An Introduction

Posted by Greg Pope

In my last post I discussed three of the traditionally defined types of validity: criterion-related, content-related, and construct-related. Now I will talk about how your organization could undertake a study to investigate and demonstrate criterion-related validity.

So just to recap, criterion-related validity deals with whether assessment scores obtained for participants are predictive of something related to the goal of the assessment. For example, if a training program conducts a four-day sales training course, at the end of which an exam is administered designed to measure trainees’ knowledge and skills in the area of product sales, one may wonder whether the exam results have any relationship with actual sales performance. If the sales course exam scores are found to be related to/predict “real world” sales performance to a high degree, then we can say that there is a high degree of criterion-related validity between the intermediate variable (sales course exam scores) and the final or ultimate variable (sales performance).

So how does one find out whether high scores on the sales course exam correspond to high sales performance (and whether low scores on the sales course exam correspond to low sales performance)? Well, within an organization there may be some “feeling” about this, for example instructors seeing star students in the course bring in big sales numbers, but how do we get some hard numbers to back this up? You will be glad to hear that you don’t need a supercomputer and a room full of PhDs to figure this out! All you need to get some data on this are some good assessment results and some corresponding sales numbers for people who have gone through the course.

The first step is to gather the sales course exam scores for the participants who took the exam. In Questionmark Perception you can use the Export to ASCII or Export to Excel reports to output in a nice user-friendly format the assessment scores for the participants who took the sales course exam. Next you will want to match the participants for whom you have exam scores with their sales numbers (e.g., how much has each salesperson sold in the last 3 months). You may want to wait a few months after these participants have taken the exam and have been out in the field selling for a while, or you could look at historical sales data if you have it. Now you put this data together into an Excel spreadsheet (or SPSS or other analysis tool if you are savvy with those tools) to analyze in way similar to this:

Next you may want to produce a scatter plot and conduct a correlation and trend line between sales course exam scores and sales dollars for the last three months:

We find the correlation is 0.901, which is very high positive relationship (people with higher sales course exam scores bring in more sales dollars). This would suggest a high degree of criterion-related validity in that the sales course exam scores do indeed predict sales performance.

To go one step further, you can take the equation produced in Excel included on the scatter plot trend line and for new sales people taking the sales course exam you can predict how much sales revenue they might bring in: y = 21049x – 3366.2 (y=estimated sales performance in dollars, x= sales course exam score). Suppose a new sales person (Rick Thomas) obtains a sales course exam score of 73%. Just plug this into the equation and y=21049(0.73)-3366.2 = $11,999.57. Voila! Based on his sales course exam score, Rick Thomas can expect to bring in about $12,000 in revenue in the next three months. With more people analyzed (we only have 10 in this example), the greater confidence one can have in the correlation coefficients obtained and the predictive equations garnered. In “real life” I would want as many points of data as possible: hundreds of salesperson data points or more.

I will focus on content validity in my next, so stay tuned!

Related posts:

1. Understanding Assessment Validity: Content Validity
2. Understanding Assessment Validity: An Introduction
3. Understanding Assessment Validity: Construct Validity