Measuring the Effectiveness of Social and Informal Learning

Posted by Julie Delazyn

How you can use assessments to measure the effectiveness of informal learning?  If people are learning at different times, in different ways and without structure, how do you know it’s happening? And how can you justify investment in social and informal learning initiatives?

The 70+20+10 model of learning – which explains that we learn 70% on-the-job, 20% from others and 10% from formal study – brings out the importance of informal learning initiatives. But the effectiveness of such initiatives needs to be measured, and there needs to be proof that people are performing better as a result of their participation in social and informal learning.

This SlideShare presentation:  Measuring the Impact of Social and Informal Learning, explains various approaches to testing and measuring learning for a new generation of students and workers.  We hope you will use it to gather some new ideas about how to answer these important questions about learning:  Did they like it? Did they learn it? Are they doing it?

 

 

Should we formalize informal learning?

Doug Peterson HeadshotPosted By Doug Peterson

Charles Jennings a champion of the 70:20:10 learning model. He explains it in this video: 70% of learning takes place through on-the-job experience, 20% of learning is through informal relationships such as mentoring and conversations with co-workers, and only 10% of learning takes place in formal settings like classrooms and eLearning courses.

I recently read an article in Learning Solutions Magazine entitled Stop Trying to Formalize Informal Learning! The author, Stephanie Ivec, makes some good points about what I would call “organic learning” (she emphasizes unofficial, unscheduled, and impromptu) and how informal learning can possibly be negatively impacted by trying to turn it into a formal process. Ms. Ivec’s position seems to be one of “let formal be formal, and let informal be informal,” which is something with which I agree – in general.

The article got me thinking about one of my favorite sayings: “Everything in Balance.” The article made me ponder what might be the right balance between formal and informal learning.

Should we try to take every informal learning experience, codify it, and teach it in a classroom or e:Learning course? Absolutely not. As a matter of fact, it would be impossible to do so in some circumstances: There are some things you are only going to learn and get better at by doing them on the job.

So then we should just let formal be formal, and informal be informal, and never the twain shall meet, right? Well, that’s where I think we need to find balance.

Consider Henry, the sales person who has been on the job for 30 years. He’s the top sales person in the company, and for good reason: He knows the company’s products like the back of his hand, he understands the company’s customer, and he has great people skills. Henry is more than willing to work with the other members of the sales team, and he provides great advice and insights to anyone who asks. He’s even taken some of the new hires under his wing for their first couple of months to “teach them the ropes.” There is some SERIOUS informal learning going on, know what I mean?

But Henry will retire in a couple of years, and he’s taking all of that (informal) knowledge with him. Somehow, that knowledge needs to be preserved within the company. This is where a bit of formalizing would be appropriate. What if Henry were asked to write a series of blog articles on his sales techniques, what he looks for in customers, how he customizes his pitch – and then new hires were required to read those articles (and take a brief knowledge check) as part of their onboarding?

Or maybe the training department could record a series of interviews with Henry, asking the right questions and capturing the right knowledge. The results could be made available in a podcast format. This means that rather than knowledge transfer taking place randomly (a junior sales person happens to be in the break room with Henry and has the courage to ask the venerable gentleman the right question), we can take some of that informal learning and add just enough of a formal wrapper to it so that we can make sure everyone benefits. We can track the learning as well as assess it.

Another concern I have about informal learning is that we don’t always know what’s being taught. Let’s say I work in a food processing plant, and at the end of each day we go through a cleaning and sanitizing process. If I mess it up, a lot of people could end up very sick. Human nature being what it is, people tend to look for shortcuts and simple ways to do things. So proper cleaning and sanitizing training is not necessarily something that should be left to on-the-job training (70%) or learning from a co-worker (20%).

Suppose my co-worker has a shortcut that appears benign and even saves time. He believes it works just as well as the proper procedures … but it doesn’t, and it’s not something that should be propagated through the workforce and to new hires. In this situation, the company might want to use observational assessments and Job Task Analysis surveys to understand what is really happening “out on the floor.” Then Formal training could incorporate the good practices and eliminate the poor ones. An alternative might be to run high-performing employees through formal training, certify their knowledge, and then designate them as “official informal trainers” back on the job: a formal/informal training hybrid.

I think there will always be, and should always be, formal, semi-formal, and completely informal learning taking place. Training organizations should support all of these, and they can help serve the company’s goals by keeping things in balance: determining what informal training should be formalized or at least captured in a formalized wrapper as a valuable learning resource.

Six trends to watch in online assessment in 2014

John Kleeman HeadshotPosted by John Kleeman

As we gear up for 2014, here are six trends I suggest could be important in the coming year.

1. Privacy. The revelations in 2013 that government agencies intercept so much electronic data will reverberate in 2014. Expect a lot more questions from stakeholders about where their results are stored and how integrity, data protection and privacy are assured, including the location and ownership of suppliers and data centres. I suspect some organizations will look to build trust with stakeholders by adopting the ISO standard on assessments in the workplace ISO 10667.

2. Anticipation of problems. Many organizations already use assessments to look forward, not just backwards. In regulatory compliance, smart organizations don’t just use assessments to check competence; they analyze results from assessments to identify trends or problems that can indicate potential issues or weaknesses, and prompt corrective measures before it gets too late. Universities and colleges increasingly use assessments to predict problems and help prevent students from dropping out (see for instance Use a survey with feedback to aid student retention). It’s exciting that assessments can be used to find issues in this way and deal with them before they happen. Don’t just treat assessments as a rear-view mirror: use them to look forward.

3. Software as a service (SaaS). For all but the very large organizations, running online assessments via a software as a service is much more cost-effective than running an on-premise system. Delegating to a service provider like Questionmark,makes the hassle of upgrading, maintaining security patches and managing deployment goes away. Increasingly, delivering assessments via a SaaS model will become the default.connected

4. Smaller and more connected world. The Internet is bringing us all together. The world is becoming connected, and in some sense smaller. We can no longer think of another continent or country as being a world away, because we can all connect together so easily. This means it is increasingly important to make your assessments translatable, multi-lingual and cross-cultural. Most medium and large organizations work across much of the world, and assessments need to reflect that.

5. Environment. I wonder if 2014 could be the year when the environmental benefits of online assessments could start to be seriously recognized. Clearly, using computers rather than paper to deliver assessments saves trees, but a bigger benefit is in reduced carbon emissions due to less traveling. For service organizations, business travel is a large proportion of carbon emissions (see for example here), and delivering training and assessments online can make a useful difference. With many countries requiring reporting of carbon emissions by listed companies, this could be important.

6. Security. Last but definitely not least, assessment security will continue to matter. As there is more awareness of the risks, everyone will expect high levels of technical and organizational security in their assessment delivery.  If you are a provider, expect a lot more questions on security from informed users; and if you are a customer or user, check that your supplier and your internal team is genuinely up to date on its security.

Read this list and look at the starting letters, and you get P – A – S – S – E – S! I wish you a happy new year and hope that each of your test-takers passes their assessments in 2014 when it is appropriate that they do so.

Announcing secure delivery of higher-stakes tests on the Apple iPad

Headshot JuliePosted by Julie Delazyn

Questionmark Secure been safeguarding the delivery of medium- and high-stakes tests for over a decade and last year became available to Mac users. Today, I’m pleased to announce that Questionmark Secure is now available for iPad users, too – and that you can download Questionmark Secure for iPad from the iTunes App store.app sotre

Like its predecessors, this free app locks down the browser, disabling functions that participants could use to print or copy exam material, ‘”accidentally” exit a test, or gain access to materials on their devices or the Internet that could give an unfair advantage. The app provides a secure environment for delivering higher-stakes assessments such as tests and exams. Used along with other measures for combating impersonation and content theft, it can help reduce the risk of cheating.

Organizations can use the app to deliver medium and high-stakes tests via low-cost, highly portable tablets — perfect for a BYOD situation or for setting up mobile test centers.

We are very pleased to offer this option to our customers, who are embracing the use of mobile devices for many different purposes.

Shenandoah University School of Pharmacy helped beta test the new app as part of an integrated mobile learning program. So far, students involved in the program have been using the Apple MacBook Pro for accessing course material and online assessments. They’ve also had their choice of using an iPod Touch, iPad 3G or iPhone for a quick mobile delivery of content. But this fall, incoming students will work with the iPad as well as the MacBook Pro. If you’d like more details about Shenandoah’s iMLearning initiative, see Shenandoah’s case study slides from this year’s  Questionmark Users Conference. (Here’s a link to the 2014 conference!) 

To learn more about the app or to download it form iTunes, click here.

Delivering a million+ assessments takes a village: A SlideShare Presentation

Headshot JuliePosted by Julie Delazyn

What does it take to deliver thousands of different assessments to thousands of students each year?

Rio Salado College, one of the largest online colleges in the United State – with 67,000 students — knows the answer: collaboration.

The people who run the college’s Questionmark assessments wear many hats. They are instructional designers, authors and programmers, as well as networking and IT services staff. Teamwork between people in these varying roles is essential. And since the college delivers more than one million assessments each year, external collaboration – with Questionmark staff – is essential, too.

A team from Rio Salado explained their cooperative approach during this year’s Questionmark Users Conference, and we’re happy to share the handouts from their presentation with you: It Takes a Village – Collaborating for Success with High-Volume Assessments.

This presentation includes an overview of how the college uses surveys, quizzes and tests within its extensive online learning programs. It also focuses on some of the many lessons gleaned from Rio Salado’s many years of involvement with Questionmark.

This is just one example of what people learn about at our Users Conferences. Registration is already open for the 2014 Users Conference March 4 – 7 in San Antonio, Texas. Plan to be there!

Test Security at Shenandoah University: A SlideShare Presentation

Headshot JuliePosted by Julie Delazyn

Students involved in Shenandoah University’s mobile learning initiative, iMLearning, use the Apple MacBook Pro for accessing course material and online assessments. They can also choose an iPodTouch, iPad 3G or iPhone for a quick mobile delivery option. Using the iPad for lab exercises, for instance, students can move from station to station without having to carry around a laptop.

In this presentation from the 2013 Questionmark Users Conference, Terra Walker and Cheri Lambert of Shenandoah University explore various security measures they use to ensure test integrity both in the iMLearning program and Windows-based testing. The presentation focuses on Using Questionmark Secure with Windows PCs and Macs but also covers measures such as question and answer randomization, seating arrangements, proctors and honor code.

This is just one example of what people learn about at our Users Conferences. Registration is already open for the 2014 Users Conference March 4 – 7 in San Antonio, Texas. Plan to be there!

Next Page »