New White Paper Examines how to Assess for Situational Judgment

Posted by John Kleeman

Is exercising judgment a critical factor in the competence of the employees and contractors who service your organization? If the answer to this is yes, as it most likely is, you may be interested in Questionmark’s white paper, just published this week on “Assessing for Situational Judgment”.

It’s not just CEOs who need to exercise judgment and make decisions, almost every job requires an element of judgment. Situational Judgment Assessments (SJAs) present a dilemma to the participant and ask them to choose options in response.


Context is defined -> There is a dilemma that needs judgment -> The participant chooses from options -> A score or evaluation is made

Here is an example: 

You work as part of a technical support team that produces work internally for an organization. You have noticed that often work is not performed correctly or a step has been omitted from a procedure. You are aware that some individuals are more at fault than others as they do not make the effort to produce high quality results and they work in a disorganized way. What do you see as the most effective and the least effective responses to this situation?
A.  Explain to your team why these procedures are important and what the consequences are of not performing these correctly.
B.  Try to arrange for your team to observe another team in the organisation who produce high quality work.
C.  Check your own work and that of everyone else in the team to make sure any errors are found.
D.  Suggest that the team tries many different ways to approach their work to see if they can find a method where fewer mistakes are made.

In this example, option C deals with errors but is time consuming and doesn’t address the behavior of team members. Option B is also reasonable but doesn’t deal with the issue immediately and may not address the team’s disorganized approach. Option D is asking a disorganized team to engage in a set of experiments that could increase rather than reduce errors in the work produced. This is likely to be the least effective of the options presented. Option A does require some confidence in dealing with potential pushback from the other team members, but is most likely to have a positive effect.

You can see some more SJA examples at http://www.questionmark.com/go/example-sja.

SJA items assess judgment and variations can be used in pre-hire, post-hire training, for compliance and for certification. SJAs offer assessment programs the opportunity to move beyond assessments of what people know (knowledge of what) to assessments of how that knowledge will be applied in the workplace (knowledge of how).

Questionmark’s white paper is written as a collaboration by Eugene Burke, well known advisor on talent, assessment and analytics and myself. The white paper is aimed at:

  • Psychometricians, testing professionals, work psychologists and consultants who currently create SJAs for workplace use (pre-hire or post-hire) and want to consider using Questionmark technology for such use
  • Trainers, recruiters and compliance managers in corporations and government looking to use SJAs to evaluate personnel
  • High-tech or similar certification organizations looking to add SJAs to increase the performance realism and validity of their exam

The 40 page white paper includes sections on:

  • Why consider assessing for situational judgment
  • What is an SJA?
  • Pre-hire and helping employers and job applicants make better decisions
  • Post-hire and using SJAs in workforce training and development
  • SJAs in certification programs
  • SJAs in support of compliance programs
  • Constructing SJAs
  • Pitfalls to avoid
  • Leveraging technology to maximize the value of SJAs

Situational Judgment Assessments are an effective means of measuring judgment and the white paper provides a rationale and blueprint to make it happen. The white paper is available free (with registration) from https://www.questionmark.com/sja-whitepaper.

I will also be presenting a session about SJAs in March at the Questionmark Conference 2018 in Savannah, Georgia – visit the conference website for more details.

GDPR: 6 months to go

Posted by Jamie Armstrong

Anyone working with personal data, particularly in the European Union, will know that we are now just six months from “GDPR day” (as I have taken to calling it). On 25-May-2018, the EU General Data Protection Regulation (“GDPR”) will become applicable, ushering in a new privacy/data protection era with greater emphasis than ever on the rights of individuals when their personal data is used or stored by businesses and other organizations. In this blog post, I provide some general reminders about what the GDPR is and give some insight into Questionmark’s compliance preparations.

The GDPR replaces the current EU Data Protection Directive, which has been around for more than 20 years. To keep pace with technology advances and achieve greater uniformity on data protection, the EU began work on the GDPR over 5 years ago and finalized the text in April 2016. There then followed a period for regulators and other industry bodies to provide guidance on what the GDPR actually requires, to help organizations in their compliance efforts. Like all businesses that process EU personal data, whether based within the U.S., the EU or elsewhere, Questionmark has been busy in the months since the GDPR was finalized to ensure that our practices and policies align with GDPR expectations.

For example, we have recently made available revised versions of our EU OnDemand service and US OnDemand service terms and conditions with new GDPR clauses, so that our customers can be assured that their agreements with us meet data controller-data processor contract requirements. We have updated our privacy policy to make clearer what personal data we gather and how this is used when people visit and interact with our website. There is also a helpful Knowledge Base article on our website that describes the personal data Questionmark stores.

GDPR

One of the most talked-about provisions of the GDPR is Article 35, which deals with data protection impact assessments, or “DPIAs.” Basically, there is a requirement that organizations acting as data controllers of personal data (meaning that they determine the purpose and means of the processing of that data) complete a prior assessment of the impacts of processing that data if the processing is likely to result in a high risk to the rights and freedoms of data subjects. Organizations will need to make a judgment call regarding whether a high risk exists to require that a DPIA be completed. There are scenarios in which a DPIA will definitely be required, such as when data controllers process special categories of personal data like racial origin and health information, and in other cases some organizations may decide it’s safer to complete a DPIA even if not absolutely necessary to comply with the GDPR.

The GDPR expects that data processors will help data controllers with DPIAs. Questionmark has therefore prepared an example draft DPIA template that may be used for completing an assessment of data processing within Questionmark OnDemand. The draft DPIA template is available for download now.

In the months before GDPR day we will see more guidance from the Article 29 Working Party and national data protection authorities to assist organizations with compliance. Questionmark is committed to helping our customers being compliant with the GDPR and we’ll post more next year on this subject. We hope this update is useful in the meantime

Important disclaimer: This blog is provided for general information and interest purposes only, is non-exhaustive and does not constitute legal advice. As such, the contents of this blog should not be relied on for any particular purpose and you should seek the advice of their own legal counsel in considering GDPR requirements.

Badging and Assessment: If they know it, let them show it!

Posted by Brian McNamara

We are delighted to announce the availability of Questionmark Badging!

With Questionmark Badging and Questionmark OnDemand, you can grant “badges” to participants based on the outcomes achieved on assessments such as certification exams, post-course tests or advancement exams.Badges associated with Questionmark assessments provide participants with portable, verifiable digital credentials.

Badges aligned with Questionmark assessments can be tied in with competencies and achievements, helping organizations provide recognition and motivation for increasing knowledge and skills. For credentialing and awarding bodies, they can increase the visibility and value of certification programs.

The new app couples Questionmark’s capabilities in delivering valid, reliable and trustworthy assessments with the industry-leading digital credentialing platform from Credly. More than just a visual representation of accomplishment, digital badges provide participants with verifiable, portable credentials that can be shared and displayed across the web, including social networking sites such as LinkedIn, Facebook and Twitter.

Find more info about Questionmark Badging right here!

GDPR is coming. Are you ready?

Posted by Julie Delazyn

Don’t get left behind as the most important change in data privacy takes effect May 2018. The new General Data Protection Regulation (GDPR) intends to strengthen and unify privacy and data protection and any organization that stores or manages data about Europeans will need to comply.

With eye-watering regulatory fines of up to €20 million or 4% of global annual turnover (whichever is greater), a credible compliance strategy is essential.

Join us for a FREE 45 minute Webinar July 26, 2017, to understand how online assessments can help you meet your GDPR challenges.

The webinar will cover:

  • What the GDPR is and who it impacts
  • Why you should care about GDPR compliance
  • How to overcome the challenges presented by GDPR — including the learning curve for your employees
  • How assessment can help mitigate GDPR risks and aid your compliance strategy
  • Considerations for implementing assessment management software to aid in compliance

We look forward to speaking to you at the webinar!

Top 5 benefits of permissions

Bart Hendrickx SmallPosted by Bart Hendrickx

We went over role-based access control and its advantages in some of my earlier blog posts:

As I mentioned previously, roles are made up of permissions. Today, I wanted to share 5 of the top benefits permissions have, in particular, Questionmark OnDemand permissions.

1. Keep your authoring content organized

You can use permissions to define where authors are allowed to create content, such as items (questions) and assessments. That enables you to set up a structure for topic folders and assessment folders, and make sure that authors won’t go outside the folders you gave them access to. That in turn helps you to keep your content organized, which benefits your assessment program’s efficiency.

2. Improve your authors’ user experience

Let’s say that all you want your authors to create are multiple choice questions. You can use a permission to that effect. As a consequence, authors will not be able to see other question types, which makes it less confusing to them. Think: I can only see what I can use. That improves their user experience.

3. Improve your reporters’ user experience

Similarly, for several reports, you can define which reports your reporters can run. By doing that, you guide your reporters to the appropriate reports and avoid confusing them with reports they should not be running.

4. Reduce error and fraud

By using permissions judiciously, you can separate duties in your organization. For example, there are permissions to create users, permissions to create assessments and permissions to schedule assessments. You can use those permissions to define that a user who can create other users cannot create assessments and vice versa. Likewise, the user who can schedule assessments cannot create users. That way, you ensure that each user is responsible for a specific part of the process. When the focus is defined, error is reduced. And if multiple users manage the process together, no-one has full power over everything, which makes fraud less likely.

5. Free up time

Finally, you can use permissions to delegate some of your role responsibilities to others. If you are the main admin user of your Questionmark OnDemand environment, you may want to give a colleague the permission to assign the reporter role to users, so that you do not have to do it repeatedly. However, you may not want to give your colleague the permission to edit the exact permissions that reporters have. By delegating the role assignment to your colleague, you remain in control over what a role can do and you get help with managing who can report. That frees up your time.

 

These are my top 5 reasons. If you can think of other benefits, contribute to this list by leaving a comment below, we’d love to see what benefits you’re experiencing.

The Power of Open: Questionmark’s open assessment platform

Posted by Steve Lay

In the beginning there was CVS, then there was SVN and now there’s Git.  What am I talking about?  These are all source code control systems, systems that are used to store computer source code in a way that preserves the complete version history and provides a full audit trail covering the who, what, when and why changes were made.

When we think of open source software we tend to think of the end product: a freely downloadable program that you can run on your computer or even a complete computer operating system in the case of Linux.  But to open source developers, open source is about more than this ‘free beer’ model of sharing software.  Open source software is shared at the source code level allowing people to examine the way it works, suggest changes to fix bugs, enhance it or even to modify it for their own purposes.  Getting the most from sharing source code requires more than just sharing an executable or a zip file of the finished product, open source developers need to open up their source code control systems too.

For years there have been services that provide a cloud-based alternative to  hosting your own source code.  The SourceForge system enjoyed many years of dominance but more recently it’s advertising sponsored model has seen it fall out of favour.

Most new projects are now created on a service called GitHub, which promises  free hosting of open source projects on a service funded by paying customers who are developing projects privately on the same platform.  The success of GitHub has been phenomenal – Google closed down its own rival service (Google Code) largely because of GitHub’s success.  In fact, GitHub is rapidly becoming a ‘unicorn’ with all the associated growing pains.  GitHub makes it easy to collaborate on projects too with its issue tracking system and user friendly tools for proposing changes (known as ‘pull requests’).

With GitHub as the de facto place to publish and share source code, it makes sense for Questionmark to use it to complement our Open Assessment Platform.  We have published source code illustrating how to use our APIs for many years and even publish the complete source to some of our connectors.  Putting new projects on GitHub means providing sample code in the most transparent and developer-friendly way possible.

Questionmark’s GitHub page lists all the projects we own.  For example, when we first brought out our OData APIs we published the sample reportlet code in the OData Reportlet Samples project.  You can experiment with these same examples running live in our website’s developer pages.

Recently we’ve gone a step further in opening up our assessment platform.  We’ve started publishing our API documentation via GitHub too!  Using a new feature of the GitHub platform we’re able to publish the documentation directly from the source control system itself.  That means you always get access to the latest documentation.

Opening up our API documentation in this way makes it easier for developers to engage with our platform.  Why not check out the documentation project.  If you’re already a GitHub user you could ‘watch’ it to get notified when we make changes.  You can even submit issues or send us ‘pull requests’ if you have suggestions for improvement.

With GitHub as the de facto place to publish and share source code, it makes sense for Questionmark to use it to complement our Open Assessment Platform.  We have published source code illustrating how to use our APIs for many years and even publish the complete source to some of our connectors.  Publishing this source code helps our customers and partners by providing working examples of how to integrate with our platform as well as providing complete transparency for our connectors allowing customers to audit the code before they run it on their own systems.  Putting new projects on GitHub means providing sample code in the most transparent and developer-friendly way possible.