New best practice webinars: Taking your assessments from to good to great

Posted by Chloe Mendonca

“Good, better, best. Never let it rest. ‘Til your good is better and your better is best.” This old little rhyme teaches us a valuable lesson: There is always room for improvement! No matter what role or business you’re in, if you’re interested in long-term success, you should strive to continuously improve your knowledge, systems and processes.

But how does this relate to assessments? Well, in many ways, there are always things we can do to develop better assessments: more secure, more trustworthy assessment programs. Maybe your current assessment program is “good”, but is “good” good enough?

We’re offering two new webinars that will help you assess how you’re currently performing in two key areas — and take your assessments from good to great:

  1. Item Writing

How to write high quality test items [35-Minute Session]

  • 3rd August, 2016, 3:00 p.m. UK BST / 10:00 a.m. US EDT

Are your items poorly written? Perhaps they’re good but you want them to be “better”. Skilfully crafted items promote learning and memory recall. They help retain knowledge, skills and/or abilities over time, but writing high-quality items isn’t as easy as it looks. This session will give you tips for taking your items to the next level.

  1. Exam Integrity

Enhancing exam integrity with online proctoring [45-Minute Session]

  • 9th August, 2016, 3:00 p.m. UK BST / 10:00 a.m. US EDT

With online proctoring rapidly gaining the attention of organisations and test sponsors around the world, many are wondering how it compares with traditional test centre proctoring. This 45-minute webinar will discuss what online proctoring is, how it works and whether it can in fact boost test security. Don’t miss this session if you’re keen to extend geographic reach and lower test administration costs.


If you’re looking to learn more about what you can achieve with Questionmark’s Assessment Management System, join our 60-minute introductory session. We’ll demo the platform live and cover a number of key features and functions. Save your seat at one of these sessions:

Intro to Questionmark’s Assessment Management System [60-Minute Session]

  • 4th August, 2016, 10:30 a.m. (BST) UK
  • 10th August, 2016, 12:00 p.m. (EDT) US

We also deliver this webinar in Spanish and Portuguese. Check out the upcoming dates and times here.

Role-Based Permissions: A How-To Guide (Part 2)

Bart Hendrickx SmallPosted by Bart Hendrickx

In my previous post on this subject (How-To Guide Part 1), I described a situation where managing permissions in the classic version of Questionmark Enterprise Manager can quickly turn into a complicated task. The new version of Questionmark, which we are starting to roll out to Questionmark OnDemand customers, offers a more efficient approach: managing permissions based on the tenets of role-based access control.

Interested in learning more about role-based permissions? Drop in on my session on this topic at Questionmark Conference 2016. Register before March 3 to take advantage of our final early-bird discounts.

The principle of role-based access control is that you use roles to define what users can do in the system. You are free to choose what a role is in your organization. You can tie it to a job title and create a role such as Learning and Development Specialist. You can map it to a role on a project team (e.g. the role of setting up a project for an employee satisfaction survey) and create a role like Project Owner. Or you can use any of the default roles that ship with the new version of Questionmark OnDemand, such as Admin and Reporter.

Roles contain permissions. For example, the Reporter role contains a set of permissions to run all reports on all results. When you add that role to a user, that user inherits those permissions. So far, this is similar to how profiles work in the classic version of Questionmark.

The power of the new role-based access control system becomes obvious when you want to give more roles to a user. In the classic version of Questionmark, you can assign only one profile to a user. In the new version, you can assign multiple roles to a user. Do you have a role for creating test items and another one for running reports, and do you have a user who will take on both roles? No problem: assign both roles to the user.

Another advantage of the new role-based access control system is that you can change the permissions of a role, which will automatically trickle down to all users who have that role. Do you want to remove the permission to run a Grade Book report from all users who have the Reporter role? Remove the permission from the Reporter role and you are done.

To ensure there are no loopholes, the new version of Questionmark OnDemand makes it impossible to assign permissions directly to users. Instead, all permissions will be granted within roles.

If you are a Questionmark OnDemand user interested in moving to the new version, contact your account manager. And if you are attending Questionmark Conference 2016, April 12-15, feel free to drop in on my session on this topic. Register before March 3 to take advantage of our final early-bird discounts.

Get tips for combatting test fraud

Chloe MendoncaPosted by Chloe Mendonca

There is a lot of research to support the fact that stepping up investment in learning, training and certification is critical to professional success. A projection from the Institute for Public Policy Research states that ‘between 2012 and 2022, over one-third of all jobs will be created in high-skilled occupations’. This growing need for high-skilled jobs is resulting in a rapid increase in professional qualifications and certifications.

Businesses are recognising the need to invest in skills, spending some £49 billion in 2011 alone on training [figures taken from CBI on skills] — and assessments are a big part of this. They have become widely adopted in helping to evaluate the competence, performance and potential of employees and job candidates. In many industries such as healthcare, life sciences and manufacturing, the stakes are high. Life, limb and livelihood are on the line, so delivering such assessments safely and securely is vital.

Sadly, many studies show that the higher the stakes of an assessment, the higher the potential and motivation to commit test fraud. We see many examples of content theft, impersonation and cheating in the news, so what steps can be taken to mitigate security risks?? What impact do emerging trends such as online remote proctoring have on certification programs? How can you use item banking, secure delivery apps and reporting tools to enhance the defensibility of your assessments?

This October, Questionmark will deliver breakfast briefings in two UK cities, providing the answers to these questions. The briefings will include presentations and discussions on the tools and practices that can be used to create and deliver secure high-stakes tests and exams.

These briefings, due to take place in London and Edinburgh, will be ideal for learning, training and compliance professionals who are using or thinking about using assessments. We invite you to find out more and register for one of these events:

 

Multilingual Approach Includes Videos

Julie Delazyn HeadshotPosted by Julie Delazyn

Questionmark customers are spread across the globe, and so it’s important for us that our product is multilingual. Here are some of the features available:

Now, we’ve added an extra resource: two new playlists on Questionmark’s YouTube channel, which feature testimonials, tutorials, overview videos and how-to’s in both Portuguese and Spanish:

Browse, watch and enjoy!

Questionmark en Espanol

Questionmark em Português

 

Measuring the Effectiveness of Social and Informal Learning

Posted by Julie Delazyn

How you can use assessments to measure the effectiveness of informal learning?  If people are learning at different times, in different ways and without structure, how do you know it’s happening? And how can you justify investment in social and informal learning initiatives?

The 70+20+10 model of learning – which explains that we learn 70% on-the-job, 20% from others and 10% from formal study – brings out the importance of informal learning initiatives. But the effectiveness of such initiatives needs to be measured, and there needs to be proof that people are performing better as a result of their participation in social and informal learning.

This SlideShare presentation:  Measuring the Impact of Social and Informal Learning, explains various approaches to testing and measuring learning for a new generation of students and workers.  We hope you will use it to gather some new ideas about how to answer these important questions about learning:  Did they like it? Did they learn it? Are they doing it?

 

 

Item Development – Five Tips for Organizing Your Drafting Process

Austin FosseyPosted by Austin Fossey

Once you’ve trained your item writers, they are ready to begin drafting items. But how should you manage this step of the item development process?

There is an enormous amount of literature about item design and item writing techniques—which we will not cover in this series—but as Cynthia Shmeiser and Catherine Welch observe in their chapter in Educational Measurement (4th ed.), there is very little guidance about the item writing process. This is surprising, given that item writing is critical to effective test development.

It may be tempting to let your item writers loose in your authoring software with a copy of the test specifications and see what comes back, but if you invest time and effort in organizing your item drafting sessions, you are likely to retain more items and better support the validity of the results.

Here are five considerations for organizing item writing sessions:

  • Assignments – Shmeiser and Welch recommend giving each item writer a specific assignment to set expectations and to ensure that you build an item bank large enough to
    meet your test specifications. If possible, distribute assignments evenly so that no single author has undue influence over an entire area of your test specifications. Set realistic goals for your authors, keeping in mind that some of their items will likely be dropped later in item reviews.
  • Instructions – In the previous post, we mentioned the benefit of a style guide for keeping item formats consistent. You may also want to give item writers instructions or templates for specific item types, especially if you are working with complex item types. (You should already have defined the types of items that can be used to measure each area of your test specifications in advance.)
  • Monitoring – Monitor item writers’ progress and spot-check their work. This is not a time to engage in full-blown item reviews, but periodic checks can help you to provide feedback and correct misconceptions. You can also check in to make sure that the item writers are abiding by security policies and formatting guidelines. In some item writing workshops, I have also asked item writers to work in pairs to help check each other’s work.
  • Communication – With some item designs, several people may be involved in building the item. One team may be in charge of developing a scoring model, another team may draft content, and a third team may add resources or additional stimuli, like images or animations. These teams need to be organized so that materials are
    handed off on time, but they also need to be able to provide iterative feedback to each other. For example, if the content team finds a loophole in the scoring model, they need to be able to alert the other teams so that it can be resolved.
  • Be Prepared – Be sure to have a backup plan in case your item writing sessions hit a snag. Know what you are going to do if an item writer does not complete an assignment or if content is compromised.

Many of the details of the item drafting process will depend on your item types, resources, schedule, authoring software, and availability of item writers. Determine what you need to accomplish, and then organize your item writing sessions as much as possible so that you meet your goals.

In my next post, I will discuss the benefits of conducting an initial editorial review of the draft items before they are sent to review committees.

Next Page »