Q&A: Sue Martin and John Kleeman discuss steps to building a certification program

Posted by Zainab Fayaz

Certification programs are a vital way of recognizing knowledge, skills and professional expertise, but, during a time of digital transformation, how do you build a program that is sustainable and adaptable to the evolving needs of your organization, stakeholders and the market?

Questionmark Founder and Executive Director, John Kleeman and Sue Martin, certification expert and Business Transformation Consultant will present the upcoming webinar Building a Certification Program in 10 easy stepsat 4pm (GMT) / 11am Eastern / 5pm CET on 12 February 2019. However, before then, we wanted to find out a little bit more about what they’ll be covering during the session, so we’ve interviewed the two experts to gain insight!

Tell us a bit about what you’ll be covering during the webinar:

Sue: During the webinar, we’ll be covering a range of things; from the conceptual steps of building a certification program to the many projects that have evolved from these and the importance of outlining key steps from the very beginning of the process for creating a comprehensive and cohesive certification program.

We will also talk about the value certification program, can add to an organization, not only in the short-haul but also for many years to come. It is important to remember, “why” and “what” you are trying to achieve, and this webinar will provide detail on how the alignment of strategic goals and communication with stakeholders contributes to the success of an adaptable certification program.

John: We’ll be discussing a range of things during the webinar, but here are the ten easy steps that we’ll be describing:

  1. Business goals
  2. Scope
  3. Security
  4. Vendor evaluation
  5. Blueprint and test design
  6. Test development
  7. Pilot
  8. Communications
  9. Delivery
  10. Reporting and monitoring

What influenced the selection of these 10-steps you have identified in building a certification program?

John:  Sue and I sat down to plan the webinar when we were together at the OEB conference in Berlin in December. Although we wanted to cover a bit some of the obvious things like test design and development, we wanted to make sure people think first about the preparation and planning, for example getting organization buy-in and working out how to market and communicate the program to stakeholders. So we’ll be focusing on what you need to do to make a successful program, and that will drive everything you do

Although you’ll be covering the key steps for building a certification program during the webinar, can you advise on three key steps you find to be the most important during the process:

Sue:
1. Planning:
The emphasis of the program’s work should be at the start, in the planning phase – especially in order to build a flexible program which will adapt to the needs of your audience and stakeholders as their needs change over time. In all of the individual project components, whether it be test creation, vendor evaluation or communications rollout, for example, design and plan for the end goal. For example, when it comes to creating an exam, you plan for it right at the start of the project – you hit the ground running! It is not all about item writing, but also the development of the project from the beginning and if you don’t plan; this can lead to the lack of validity in the exam program and inconsistency over time

2. Practical tips and tricks for approaching various elements of your program development: It is important to set out the target audience; identify their learning journey and how they learn – in knowing this, can you go forward and build a certification program that can become integrated and aligns with the learning process

3. Scope: This is very important; setting the scope is a priority. Of course, in the greater scheme of things; you’ll have a mission statement, which provides you with a strategic vision, but when it comes to the finer detail and knowing what countries to enter, the pricing structure or knowing whether to offer remote proctoring; always keep in mind three things: the value contribution, the stakeholders and ask yourselves “yes, but why?”; as this will help align with organizational objectives.

What can attendees take away from the webinar you’ll present?  

Sue: Those attending will learn the value and importance of planning and questioning everything from the start of the process. We’ll share advice on the importance of having a value statement for every part of the process and making sure you know that a certification program is what you are looking for. By attending you can walk away with knowing the operational and strategic steps you must go through in order to build a program that is sustainable; think of it as a checklist!

John: If you’re starting a new certification program, I think this webinar will help guide you and help you create it more easily and more effectively. And if you already have a certification program and want to improve it, you’ll probably be doing a lot of what we suggest already but I hope they’ll be something for everyone to take away and learn.

Want to know more?

If you’re interested in learning more about the steps to building a certification program that meets the needs of your organization and stakeholders; then join John and Sue deliver a webinar session dedicated to Building a Certification Program in 10 easy steps on 12 February 2019 at 4pm (GMT) / 11am Eastern / 5pm CET.

You can register here.

A little bit more about our two experts:

John Kleeman is Executive Director and Founder of Questionmark. He has a first-class degree from Trinity College, Cambridge, and is a Chartered Engineer and a Certified Information Privacy Professional/Europe (CIPP/E). John wrote the first version of the Questionmark assessment software system and then founded Questionmark in 1988 to market, develop and support it. John has been heavily involved in assessment software for 30 years and has also participated in several standards initiatives including IMS QTI, ISO 23988 and ISO 10667. John was recently elected to the Association of Test Publishers (ATP) Board of Directors.

Sue Martin is a trusted advisor to companies and institutions across Europe in the area of workforce credentialing, learning strategies and certification. Her career prior to consulting included a role as Senior Global Certification Director for SAP and several regional and global management roles in the testing industry. She has also held several positions within industry institutions, such as the Chair of the European Association of Test Publishers and is currently a member of the Learning & Development Committee at BCS (British Computer Society).

 

International University gains scalability, advanced authoring, single sign-on and more!

ChloePosted by Chloe Mendonca

Earlier this month I had the pleasure of speaking with The International University of Languages and Media (IULM) based out of Milan, Italy. IULM has been using Questionmark’s Assessment Management System for almost 10 years, so I was also keen to see what prompted their recent move from managing their assessment program on-premise to Questionmark OnDemand (Questionmark’s SaaS cloud-based solution).iulm

IULM serves approximately 5,000 students enrolled across a range of postgraduate and undergraduate courses including interpretation, translation and cultural studies; communication, public relations and advertising, and arts and tourism.

Patrizia Lettieri, the digital platform administrator in the university’s IT department explains why after 10 years, the IULM decided to make the move:  “Our teachers liked using Questionmark; we didn’t want to change that, but there was definitely a need for a more scalable and innovative solution that would match the direction the university was headed in terms of growth. Security and reporting were also very important to us and there is nothing on par with Questionmark OnDemand in regards to security and reporting.”

Here are some highlights from our conversation. You can read the entire case study here.

Advanced authoring
A big draw for IULM in using Questionmark was the ease of use when it came to assessment authoring and improving the way items are written and also the efficiency with which they can manage their item bank. Patrizia elaborates: “The variety of item types and ability to embed images, videos and links into our questions and feedback enable us to further develop the way we are helping our students to learn. Meta tagging of items is also very simple. Before using OnDemand we had to separate these into different folders and our teachers used to delete items that were invalid. Now we can easily assign the appropriate meta tags and modify the status of questions to retired if they are invalid.”

Scalability and save-as-you-go
iulm_campusBefore using Questionmark OnDemand IULM had a few challenges when it came to assessing large numbers of students.  With OnDemand, IULM can reliably deliver their assessments to multiple classes of students. And in the event that something does go wrong — such as a power cut or other technical failure — Questionmark’s save-as-you-go is there to ensure student answers are automatically saved at regular intervals. “My priority is the students; they are our customers and I want them to be comfortable,” adds Patrizia. “With OnDemand everything runs smoothly and our students are more comfortable and confident when taking their assessments.”

Single Sign-On
It used to be that every day, staff and students at the university were having to interact with a variety of applications requiring individuals to remember different credentials for the various applications.  “With Questionmark OnDemand, we were able to take advantage of single sign-on (SSO). For a forward-thinking technological university, improving access for both students and staff just made sense for us.” SSO also saves IULM time on the administrative side. Once students have completed their studies and leave the university, staff no longer have to remove their access to each application but only need to remove each student’s identity provider account. Without that account, the student can no longer log on to any of the linked applications.

Multilingual delivery
Being an international university, having students who speak different languages is a common scenario. Questionmark’s multilingual delivery helps IULM accommodate students whose first language is not Italian or English by enabling them to change the delivery interface to their preferred language. Having the navigation controls in a participant’s own language can often make students feel more at ease during the test taking experience.

If you’re interested in trying out Questionmark OnDemand, request a demo today and one of our team will show you how simple assessment management can be.

Security, Reporting and Online Testing in Academia: A Q&A with Don Kassner

Julie ProfileI recently spoke with Don Kassner, who has joined Questionmark as vice president of academic markets. I wanted to take a moment to welcome Don and to ask him a few questions about his extensive background, his new job role and his hopes for the future. Here’s a snippet of our conversation:

Don Kassner, VP of academic markets, Questionmark

You have had an extensive career. What is your background, and how has it influenced the insight you have on the learning and assessment world?

At heart, I am an entrepreneur.  I started my first business when I was 8 years old and have been starting and building things ever since. In the last 15 years or so I have focused on technology, training and education. During that time, I have served as the CFO for an auto dealership training company and as president of a small university in addition to founding and building the largest online proctoring company. I have also served on the faculty of San Jose State University (economics) and have been an active member of multiple onsite accreditation evaluation teams.

Now that you are part of the Questionmark team, can you share a little about your new role and the goals you have for this position?

I am excited to reach out to the academic markets and address ongoing concerns around testing.

Colleges and universities want to deliver tests and exams that are consistent, fair, reliable and defensible. They want to deliver and monitor course evaluations, identify knowledge gaps and place students in appropriate courses, and be able to analyze questions to determine which ones are valid and fair. They want to do this while increasing student satisfaction and reducing cheating.

With Questionmark’s extensive security, reporting and analytics tools, that’s all possible.  My focus is to leverage my insight and experience and put this secure and easy-to-use tool in the right hands.

Can you share some of the important topics surrounding the academic markets that are on your radar?

In academia, there are real issues related to online testing. These are the top two:

  • Is the student doing the work? – With technology comes flexibility and as the stakes become higher, students will naturally look for ways to enhance their scores.
  • Are the exam results fair?  Did all the students face the same conditions with the same opportunities and were the exam rules followed?

Questionmark OnDemand, especially its analytics and reporting tools, can provide reports that can uncover the meaning hidden within assessment results, such as an item or assessment’s reliability and defensibility. In regards to security, Questionmark Secure’s lock-down browser plays a huge role in helping organizations provide a secure environment in which to deliver high stakes assessments. This can significantly help reduce the risk of cheating when deployed along with other defenses to combat impersonation and content theft.

We’re very excited to have you on board! What are your plans for keeping in touch with us on The Questionmark Blog?

Once a teacher, always a teacher! I look forward to turning back the clock to my university days and speaking on the hot-button topics that surround academic testing. My experience in education and entrepreneurship will help me as I share my thoughts on current market trends and the future of assessment and testing.

Thank you Don for taking the time to speak to us today!

You can follow Don on Twitter and connect with him on LinkedIn

Online Proctoring: FAQs

John Kleeman HeadshotPosted by John Kleeman

Online proctoring was a hot-button topic at Questionmark’s annual Users Conference. And though we’ve discussed the pros and cons in this blog and even offered an infographic highlighting online versus test-center proctoring, many interesting questions arose during the Ensuring Exam Integrity with Online Proctoring  session I presented with Steve Lay at Questionmark Conference 2016.

I’ve compiled a few of those questions and offered answers to them. For context and additional information, make sure to check out a shortened version of our presentation. If you have any questions you’d like to add to the list, comment below!

What control does the online proctor have on the exam?

With Questionmark solutions, the online proctor can:

  • Converse with the participant
  • Pause and resume the exam
  • Give extra time if needed
  • Terminate the exam

What does an online proctor do if he/she suspects cheating?

Usually the proctor will terminate the exam and file a report to the exam sponsor.

What happens if the exam is interrupted, e.g. by someone coming in to the room?

This depends on your security protocols. Some organizations may decide  to terminate the exam and require another attempt. In some cases, if it seems an honest mistake, the organization may decide that the proctor can use discretion to permit the exam to continue.

Which is more secure, online or face-to-face proctoring?online proctoring

On balance, they are about equally secure.

Unfortunately there has been a lot of corruption with face-to-face proctoring, and online proctoring makes it much harder for participant and proctor to collude as there is no direct contact, and all communication can be logged.

But if the proctors are honest, it is easier to detect cheating aids in a face-to-face environment than via a video link.

What kind of exams is online proctoring good for?

Online proctoring works well for exams where:

  • The stakes are high and so you need the security of a proctor
  • Participants are in many different places, making travel to test centers costly
  • Participants are computer literate – have and know how to use their own PCs
  • Exams take 2-3 hours or less

If your technology or subject area changes frequently, then online proctoring is particularly good because you can easily give more frequent exams, without requiring candidates to travel.

What kind of exams is online proctoring less good for?

Online proctoring is less appropriate for exams where:

  • Exams are long and participants needs breaks
  • Exams where participants are local and it’s easy to get them into one place to take the exam
  • Participants do not have access to their own PC and/or are not computer literate

How do you prepare for online proctoring?

Here are some preparation tasks:

  • Brief and communicate with your participants about online proctoring
  • Define clearly the computer requirements for participants
  • Agree what happens in the event of incidents – e.g. suspected cheating, exam interruptions
  • Agree what ID is acceptable for participants and whether ID information is going to be stored
  • Make a candidate agreement or honor code which sets out what you expect from people to encourage them to take the exam fairly

I hope these Q&A and the linked presentation are interesting. You can find out more about Questionmark’s online proctoring solution here.

How to transform recruitment and hiring with online testing

Julie Profile

Posted by Julie Delazyn

Did you know that 80 percent of employee turnover is a result of poor hiring practices? With turnover costs as high as $20,000 per exiting employee, improving the screening and testing process is a priority for every HR professional.

Online testing is making the hiring process easier and more effective. In this webinar,  “How to transform recruitment and hiring with online testing,” we’ll also share best practices and a case study from Canada Post Corporation.

Glen Budgell_large

Dr. Glen Budgell, PhD and senior strategic HR advisor at Human Resource Systems Group (HRSG)

 Join us on June 14 at 12:00 p.m. EDT. Click here to reserve your spot!

The webinar is designed to give you an up-close look at the latest assessment technologies. To learn more, I spoke to Dr. Glen Budgell, PhD and senior strategic HR advisor at Human Resource Systems Group (HRSG), who will be leading the session. Glen has over thirty years of experience managing the promotion, development and delivery of consulting services in the area of assessment and testing.

Here is a snippet from my conversation with Glen:

How can online testing transform recruitment and hiring?

Online testing has transformed the recruitment and hiring landscape in many ways. The days in which employers were limited to local talent is over: online tests can be administered at the click of a button to job candidates in remote regions, thereby increasing both the diversity of the talent pool and the inclusiveness of the work place. Standardized testing (whether paper-pencil or computer-based) provides job candidates with a sense of fairness in the hiring process.  When you combine this sense of fairness with an innovative online test that is both job relevant and engaging, you really “wow!” the job candidate, and that is one of our goals at HRSG: to wow our clients and their job candidates with innovative employment tests.

Online testing platforms such as Questionmark provide employers and firms like HRSG the tools to efficiently develop and deliver assessments. At HRSG, we use Questionmark to develop employment tests for our clients so that they can identify the right talent faster, and at a lower cost.

What are some best practices that are important to developing a strong assessment process?

Traditional testing best practices are always relevant, and we strongly urge our clients to conduct the required job/task analysis before choosing or building an employment test. In addition, and especially if the test is being built in house, it should be ensured that the test is both job relevant and linked to the abilities/skills identified in the job analysis.

Of course, online / computer-based testing has introduced a myriad of other challenges. Companies must ensure that the testing software they use is secure and candidate data is protected. We rely on the expertise of Questionmark to provide our clients with the required security and data protection features mandated by the ISO and ICT standards. Furthermore, we share with our client’s methods and best practices for cheating prevention and analysis, monitoring the web for leaked test content, and most importantly, providing candidate’s with technical support and accessibility features when required.

What tools will attendees walk away from the webinar with:

We’ll also discuss:

  • When to buy an off the shelf assessment when to build in house
  • How to enhance the candidate experience with online testing
  • How to develop and use innovative item types
  • Importance of candidate perceptions of online testing
  • Best practices for unproctored internet testing
  • Case study which describes best practices with high volume testing

Thank you, Glen, for taking the time to speak to us!

We look forward to seeing you at this complimentary webinar on Tuesday, June 14 at 12:00 p.m. EDT.

Click here to register and save your spot.

Establishing a data-driven assessment strategy – A Q&A with Amazon

Headshot Julie

Posted by Julie Delazyn

Jason Sunseri is a senior Program Manager – Learning Technology at Amazon. He will be leading a discussion at Questionmark Conference 2016 in Miami, about Creating a Global Knowledge and Skills Assessment Program for Amazon Sellers.

Jason Sunseri, Program Manager – Learning Technology, Amazon

Jason’s session will look at how Amazon Seller Support and Questionmark OnDemand have partnered to deliver a world-class solution. Jason will illustrate how Amazon has used the OnDemand platform to deliver a robust, data-driven assessment strategy.

I recently asked him about his session:

Tell me about Amazon and its use of assessments:

Amazon Seller Support engages with the 2.5 million+ global sellers represented on the Amazon platform. Due to rapid global expansion across the platform, the Amazon Seller Support needed to find a technology and assessment partner that could support both its knowledge and skill acquisition assessment strategies.

How does Amazon use data to drive strategy?

Assessments play a huge role at Amazon. We have really evolved into a data-driven culture and we use assessments in surveys and inside curriculum to assess training and performance, and to identify early issues and trends in order to tweak training content and fix errors.

What role does Questionmark play in that strategy?

We rely heavily on reports — Survey Matrix, Job Task Analysis and other report functions — to assess performance. We’re able to leverage the tool by having individual training centers analyze learning and training gaps and pass on those results. It allows us to see how and why a site is succeeding; where that behavior stems from — it’s really cool to see.

What are you looking forward to at the conference?

It’s Miami, so…the weather, for sure! In all seriousness, I look forward to learning about how other Questionmark users utilize the same tools and how their approach varies from ours.

Thank you, Jason for taking time out of your busy schedules to discuss your session with us!