Into the third decade of Internet assessment

john_smallPosted by John Kleeman

As I take stock of the past year and look forward to the next, I’m doing so with great appreciation for all of Questionmark’s customers and users and for all my colleagues. I’d like to extend my best wishes to everyone for 2010.

I’m looking forward to the New Year as the start of a decade that promises to be an exciting one in which to work.There should be some amazing new developments during the 2010s as the Internet—which I regard as the third great invention of humankind—continues to transform assessment!

I’d suggest the first great invention was writing, allowing people to store information and knowledge and pass it on to others. Before writing, information could only be transferred by songs and ballads, or one person’s advice to another. Once writing was invented, knowledge could be written and stored, and more complex societies could start to form.

The next great invention was the printing press. Books and knowledge were not just restricted to the rich or the learned but could be communicated to all. Access to information and ideas became much easier, and this led in time to equality and freedom and our modern world.

The Internet, by bringing everyone together irrespective of geography and allowing the synergy of the crowd, rather than just from the writer to the reader, is transforming the world and society.  These are exciting and interesting times to be living and working. We don’t yet know where the Internet will take us, but it already is changing our world and bringing it together.

Working in the field of assessment, we are fortunate to be enablers for the Internet revolution. The Internet means that knowledge, learning and ideas spread faster than ever before, and we are freed from the constraints of geography. You can read this blog entry from Kansas to Karachi, at the time you want. And you can comment on it, tweet it, or ignore it and move to something better, all in just a few keystrokes.

Assessment contributes to the Internet and the Internet contributes to assessment. The Internet is about learning and assessment is the cornerstone of learning: it diagnoses what you know and need to learn, helps confirm what you have learned and helps personalize your learning path.  The promise of the Internet is that a child in the furthest corner of the world can learn from the greatest teacher, and that the potential of everyone in the world can be fulfilled. By effective use of surveys, quizzes, tests and exams, we can be part of making this happening.

The Internet also contributes to making assessment better. From online item writing workshops and item review with tools such as Questionmark Live, to delivery over the Internet and on mobile devices, and to passing and confirming assessment results online, the Internet changes every part of assessment. And this can only change and change for the better as the technology becomes more reliable and the demands for global assessment increase.

The 1990s saw the beginnings of Internet assessment (with Questionmark proud to have produced the first Internet assessment product in 1995) and the 2000s have seen Internet assessment become widespread and useful. Questionmark has an exciting announcement to make about a new version of our software in January 2010 that will make mobile assessments and assessments in frames within wikis, blogs and portals much, much easier. With news like that for starters, the 2010s look a very exciting time for Internet assessment.

Webinar: Integrating Questionmark Perception with SAP Learning Solution

Joan Phaup

Posted by Joan Phaup

I hope everyone is enjoying the holiday season and anticipating a peaceful and progressive New Year!

One event we are looking forward to is a Customers Online Web Seminar to be presented at 1 p.m. Tuesday, January 26th, by Timothy Pospisil of the Nebraska Public Power District (NPPD).  Timothy will explain how NPPD, the largest utility in the nation’s only all-public-power state, has integrated Questionmark Perception with SAP Learning Solution to streamline assessment and training.

Working within a highly-regulated industry, NPPD requires rigorous training and sound assessment of employees’ knowledge and competencies.  The company wanted to achieve all this while providing a “one-stop learning shop” that would put employees in control of scheduling their own training and tracking their own progress.  At the same time, NPPD wanted to give supervisors and managers oversight of their direct reports, training plans and progress.  The one-stop shop, which makes it possible to manage computer-based training in the same system as final records, also requires  robust, secure online exams that utilize a wide variety of question types.

The integration of Perception and SAP has greatly enhanced the training process for employees, particularly Nuclear Plant workers.  Participants in this Web seminar will learn how this integration meets the strict requirements of a nuclear training program, understand the synergy between Perception and SAP, and find out what NPPD has learned as the first organization to integrate the two systems.

Prashanth Padmanabhan, Product Manager, SAP Enterprise Learning will provide a brief overview of the SAP/Questionmark integration.  Consultants Jason Ichen and Van Vayntrub, who assisted NPPD with the implementation of SAP, will be available on the call to answer questions about the process.

Click here to sign up for this complimentary seminar.

Improved Shared Question Sets in Questionmark Live


Posted by Jim Farrell

One of goals of the Questionmark Live team is to constantly improve this browser-based authoring tool by releasing new functionality on a regular basis. In our latest release we have made it easier to quickly share a question set with a group of people, including those who are not yet members of the Questionmark Live community.

Watch the video below to see the improvements to shared question sets!


Understanding Validity: More on Construct Validity


Posted by Greg Pope

In my last post I introduced construct validity. Here I will talk more about some specific aspects of construct validity.

I previously mentioned “unidimensionality,” and there may be some people out there who would like to know what this terms means. In the assessment realm, unidimensionality refers to the measurement of one psychological dimension/trait/construct/attribute/skill/ability/etc. For example, if we have an assessment that is designed to measure the construct “math ability” we would want to ensure that all the questions in the assessment measure this construct and only this construct. One can investigate the degree to which an assessment is unidimensional by using some well known statistical analysis methods such as principal component analysis (PCA) or factor analysis (FA) to confirm/explore what dimensions each question in the assessment loads onto. I have done a few of these in my day using software like SPSS and I can tell you from firsthand experience, they are a ton of fun.

Unidimensionality is an important assumption in a number of areas of psychometrics and has implications for statistics like internal consistency reliability (e.g., Cronbach’s Alpha will be maximized when all items are measuring the same construct) and the interpretation of participant scores (e.g., if an assessment is measuring a random smattering of dimensions then what does a participant score really mean?). Here’s how this concept of unidimensionality fits in with the concept of construct validity: If an assessment is designed to measure only one dimension/construct, this should actually be the case, and this design assumption can be investigated/confirmed. If an assessment is composed of four topics, each of which is designed to measure only one dimension, then this can be investigated/confirmed. In other words, the dimensionality assumptions about the measurement of the construct(s) composing the assessment should be validated, for example using methods like PCA or FA.

Moving on to the details of convergent and discriminant validity, these two things are in some ways flip sides of the same coin. With convergent validity, we are seeing whether our math ability test correlates with other well known math ability tests that are being used. With discriminant validity, we are seeing whether our math ability test does not correlate with well known tests out there that measure something different than math ability, like verbal ability.

Using our trusty statistical analysis program, Excel, we can conduct a mini convergent and discriminant validity study. We had the same ten participants take four math ability tests for the convergent validity study: Our math ability test and three other well known tests out there. For the discriminant validity piece we also had the same ten participants take three well known verbal ability tests. To investigate convergent validity we correlated the assessment scores from Our math ability test with scores obtained from the three other math ability tests and we found high correlations with each, for example the correlation between Our math ability test and the EU math ability test is 0.966, the correlation between Our math ability test and the Australian math ability test was 0.960, and the correlation between Our math ability test and the US math ability test was 0.962:

Greg Pope image001

To investigate discriminant validity we correlated the assessment scores from Our math ability test with scores obtained from the three verbal ability tests and we found very low correlations with each, for example the correlation between Our math ability test and the Canadian verbal ability test was 0.008, the correlation between Our math ability test and the US verbal ability test was -0.053, and the correlation between Our math ability test and the British verbal ability test was -0.078: 

gret pope image00 078

In terms of nomological validity, this is something that needs to be addressed in situations where new research is being conducted into an existing construct or new construct. We would want to ensure that current research is supported by previous research. If a completely new construct is being proposed we would need to justify how this construct is similar and different to other construct research that has been done.

In my next post I will discuss modern perspectives on validity.

Podcast: Using Questionmark Live for Item Writing Workshops

Posted by Joan Phaup

The School of Justice at Miami Dade College in Florida plays an important role in helping people prepare for careers in public safety. Not only does the school offer degree programs, it is also home to the state’s basic abilities testing program for prospective police, correctional and probation officers.

The Florida Basic Abilities Test (F-BAT), which measures basic abilities of recruits planning to enter training programs, needs to be kept current and relevant. The College reviews the test annually and freshens the questions, drawing on the expertise of subject matter experts (SMEs) from all over the state.

The introduction  this year of Questionmark Live has given the college a new, more efficient  way of working with SMEs: During a recently item writing workshop held at the College, SMEs were given a quick tutorial on the use of Questionmark Live and were soon creating questions using its simple browser-based authoring tools. Their efforts were so successful  that the college plans to run future workshops at a distance, doing away with the need for SMEs to meet together in order to help create test content.

You can learn more about the college’s role here and find out more about the item writing sessions by listening to the following podcast with Lebsica Gonzalez, F-BAT project manager at Miami Dade College.

Understanding Assessment Validity: Construct Validity


Posted by Greg Pope

In my last post I discussed content validity. In this post I will talk about construct validity. Construct validity refers to whether/how well an assessment, or topics within an assessment, measure the educational/psychological constructs that the assessment was designed to measure. For example, if the construct to be measured is “sales knowledge and skills,” then the assessment designed to measure this construct should show evidence of actually measuring this “sales knowledge and skills” construct.

It will come as no surprise that measuring psychological constructs is a complicated thing to do. Human psychological constructs such as “depression,” “extroversion” or “sales knowledge and skills” are not as straightforward to measure as more tangible physical “constructs” such as temperature, length, or distance. However, luckily there are approaches which allow us to determine how well our assessments accomplish the measurement of these complex psychological constructs.

Construct validity is composed of a few areas with convergent and discriminant validity being the core:

validity 7In my next post I will drill down more into some of these areas of construct validity.