Posted by Chloe Mendonca
Our next European Breakfast Briefing will focus on role of assessments in regulatory compliance and mitigating risks.
The briefing, to be presented by Questionmark and Stoas Learning, will take place the morning of Tuesday, 9 October, in Utrecht, Netherlands.
It will concentrate on these topics:
- Using assessments to identify knowledge gaps
- How to determine if employees understand laws and regulations
- Using mobile smartphones and tablets for mobile assessments
- Supporting assessments in multiple languages
- Observational assessments
- Embedding assessments in wikis, portals and blogs
- Examples from financial services organizations that use assessments to mitigate risk and document compliance
This briefing will of particular interest to people involved in compliance and/or risk management, learning and development, and human resources.
Join us for breakfast and a morning of learning and networking. You will find details and free registration here.
Posted By Doug Peterson
At this point in the design process, you’ve written all the items for your assessment. Before you assemble them into a test, they need to be reviewed. Be sure to link each item to the Test Content Outline (TCO), then ask a group of subject matter experts (SMEs) to review the questions. This very well could be the same group of SMEs that wrote the questions in the first place, in which case they can simply review each other’s work. There are three main things to look at when reviewing each item:
- Spelling and grammar
- Clarity – is it clear what the item is asking? Is the item asking only one question and does it have only one correct answer? Is the item free of any extraneous information, bias, and stereotyping?
- Connection to TCO – it is legitimate to include this item on this assessment because it clearly and directly pertains to the goals of the training.
Once you are confident that you have a complete set of well-written items that tie directly to your TCO, it’s time to start putting the assessment together. In addition to determining which questions from your item bank you want to include (which will be discussed in the next entry in this series), you must also develop test directions for the participant. These directions should include:
- Purpose of the assessment
- Amount of time allowed
- Procedures for asking questions
- Procedures for completing the assessment
- Procedures for returning test materials
As part of your participant directions, you may want to consider including sample items, especially if the format is unusual or unfamiliar to the participants. Sample items also help reduce test anxiety. Remember, you want to assess the participant’s true knowledge, which means you don’t want a “stress barrier” getting in the way.
In addition to the participant’s instructions, you also want to put together instructions for the assessment administrator – the instructor or proctor who will be handing the test out and watching over the room while the participant’s take the assessment. Having a set of written instructions will help ensure consistency when the assessment is given by different administrators in different locations. The instructions should include:
- The participant’s instructions, which should be read aloud
- How to handle and document irregularities
- The administrator’s monitoring responsibilities and methods (e.g., no phone conversations, walk around the room every 10 minutes, etc.)
- Hardware and software requirements and instructions, if applicable
- Contact information for technical help
As you develop your assessment, make sure that you are taking into account any local or national laws. For example, American test centers must comply with the Americans with Disabilities Act (ADA). The ADA requires that the test site be accessible to participants in wheelchairs and that compensation be made for certain impaired abilities (e.g., larger print or a screen reader for visually impaired participants). The administrator’s instructions should cover what to do in each case.
Posted by Julie Delazyn
If people already know something well, then teaching them about it is a waste of resources and motivation. That’s where “testing out” comes in. Diagnostic tests can provide a way of identifying what employees know and determining if they need further training.
Offering training needs assessments enables employees who can demonstrate that they already understand a subject to “test out” of unnecessary training on it. This saves time for both training departments and employees. And, of course, that translates into cost savings, too. At the Questionmark 2011 Users Conference, for instance, a large accountancy firm reported saving more than 500 hours of training thanks to a diagnostic test, with savings of more than US$40,000.
Is testing out a supported practice? A Training Room article by Meg Sczyrba in the American Banking Association Bank Compliance magazine reported a consensus among several US regulators about the acceptability of testing out, provided the testing program is well-structured.
To read more about effective ways to use diagnostic and other types of assessments in the context of compliance, check out our white paper: The Role of Assessments in Mitigating Risk for Financial Services Organizations. You can download it free here, after login.
Posted by Julie Delazyn
To use assessments effectively, it’s important to understand their context and uses within the learning process.
Last week I wrote about needs assessments, and today I’ll explore reaction assessments.
- Determining the satisfaction level with a learning or certification experience
- Gathering opinions from learners about course materials, instructors, learning environments, and so forth
- Identifying shortcomings of a learning experience in order to help improve it for others
- Aiding the planning process for revising a course and/or the way in which it is delivered
- Level 1 evaluations (as per Donald Kirkpatrick)
- Course evaluations
- Smile sheets/ happy sheets
- Opinion surveys
Answers to Question 8, analyzed below, reveal at that respondents feel they have sufficient time for the training they need to do their jobs well. But their answers to Question 9 — indicating that many people had problems with the timing of training courses — prompted their company to revise its training schedule.
For more details about assessments and their uses check out the white paper, Assessments Through the Learning Process. You can download it free here, after login. Another good source for testing and assessment terms is our glossary.
In my last post in this series, I will take a look at summative assessments.
Posted by Joan Phaup
I’m eagerly looking forward to the keynote presentation Charles Jennings will deliver at the Questionmark 2013 Users Conference on The Challenge of Measuring Informal and Workplace Learning.
Charles is one of the world’s leading thinkers and practitioners in learning and development — currently head of Duntroon Associates and previously Chief Learning Officer for Reuters and Thomson Reuters.He will be talking at the conference about how the 70:20:10 learning framework — based on studies that show high performers learn approximately 70% from experience, 20% from others and 10% from formal study – is being adopted by many organizations around the world.
The keynote will address how this framework serves as a strategy for extending development beyond formal, structured learning to include informal and experiential learning opportunities.
I spoke with Charles recently and asked him for some details about his presentation. For example:
- How would you describe the 70:20:10 framework?
- What are the key challenges of measurement and evaluation within that framework?
- How will your conference presentation address those kinds of challenge?
- What advice would you give to organizations that want to use online assessments to measure the effectiveness of informal and experiential learning?
If you’d like to find out how he answered, listen to this podcast or read the transcript. There will be much, much more, of course, in his keynote address and in the conference program, which we are busy planning right now.
Early-bird registration savings are available through November 16 — so keep an eye on the conference website and be sure to register soon! We’ll look forward to seeing you in Baltimore, Maryland, March 3 – 6 at this terrific learning and networking event.
Posted by John Kleeman
Almost every day, you can open the financial pages of a newspaper and see a new compliance failure – another company fined multi-millions for breaking regulations. How can an organization develop a culture of compliance?
Charles Jennings, former CLO of Reuters and our keynote speaker at the 2013 Questionmark User Conference, made an interesting comment in a recent blog post:
There seems to be a common thread that runs through almost all high-profile compliance catastrophes. It is that the top-tier executives and middle managers in the organisations simply didn’t model the behaviours that would lead to a culture of compliance.
In other words, an organization must not just pay lip service to complying with regulations but must also communicate effectively to its employees that it really means it. This is a regular theme from regulators. The UK Ministry of Justice, in its guidance on the UK Bribery Act, lists Top-level Commitment as one of its six key principles for bribery prevention. And the U.S. Department of Justice says in its Principles of Federal Prosecution of Business Organizations:
Prosecutors should therefore attempt to determine whether a corporation’s compliance program is merely a “paper program” or whether it was designed and implemented in an effective manner. … prosecutors should determine whether the corporation’s employees are adequately informed about the compliance program and are convinced of the corporation’s commitment to it.
Obviously, commitment is commitment, and you can’t fake it. Executives and managers need to genuinely believe that compliance with regulations is important and exhibit appropriate behaviour.
But my research about this issue tells me that by administering regular tests, an organization can reinforce the message that it is committed to compliance – and that this works best under the following five conditions:
- Employees are required to take tests regularly
- Employees believe the tests are fair and genuinely measure their ability to do a job or understand and apply regulations
- Questions are seen to be relevant, not just a tick-the-box exercise – for instance by having employees respond to real-life scenarios
- There are consequences for repeated failures
- Managers and executives set an example by taking relevant tests themselves
Do regulators agree? Here is a quote from the UK Financial Services Authority about an insurance company fined UK£5.5m for bribery (my emphasis):
Aon Ltd should have ensured that appropriate members of staff – particularly those in the Aviation and Energy divisions – received focused training in relation to this area and were tested on their understanding of the relevant risks involved. Effective training and testing in this regard would have emphasised to staff the importance of carrying out effective due diligence prior to authorising an Overseas Third Party for payment.