How can you assess the effectiveness of informal learning?

Posted by John Kleeman

Lots of people ask me how you can use assessments to measure the effectiveness of informal learning.  If people are learning at different times, in different ways and without structure, how do you know it’s happening? And how can you justify investment in social and informal learning initiatives?

The 70+20+10 model of learning is increasingly understood – that we learn 70% on-the-job, 20% from others and 10% from formal study. But as people invest in informal learning initiatives, a key question arises. How do you measure the impact? Are people learning? And more importantly, are they performing better?

Did they like it? Did they learn it? Are they doing it?In a presentation at the Learning Technologies conference in London in January, I suggested there are three areas in which to use assessments:

Did they like it?

You can use surveys to evaluate attitudes and reactions – either to specific initiatives or to the whole 70+20+10 initiative. Measuring reaction does not prove impact, but yields useful data. For example, surveys yielding consistently negative results could indicate initiatives are missing the mark.

You could also look at the Success Case Method, which lets you home in on individual examples of success to get early evidence of a learning programme’s impact. See here and here for my earlier blog posts on how to do this.

Of course, if you are using Questionmark technology, you can deliver such surveys embedded in blogs, wikis or other informal learning tools and also on mobile devices.

Did they learn it?

There is strong evidence for the use of formative quizzes to help direct learning, strengthen memory and engage learners. You can easily embed quizzes inside informal learning, e.g. side by side with videos or within blogs, wikis and SharePoint, to track use and understanding of content.

With informal learning, you also have the option of encouraging user-generated quizzes. These allow the author to structure, improve and explain his or her knowledge and engage and help the learner.

You can also use more formal quizzes and tests to measure knowledge and skills. And you can compare someone’s skills before and after learning, compare to a benchmark or compare against others.

Are they doing it?

Of course, in 70+20+10, people are learning in multiple places, at different times and in different ways. So measuring informal learning can be more difficult than measuring formal, planned learning.

But if you can measure a performance improvement, that is more directly useful than simply measuring learning. A great way of measuring performance is with observational assessments. This is described well in Jim Farrell’s recent post Observational assessments- measuring performance in a 70+20+10 world.

To see the Learning Technology presentation on SlideShare, click here. For more information on Questionmark technologies that can help you assess informal learning, see www.questionmark.com.

Measuring learning in SharePoint: where to find info

Posted by Julie Delazyn

The way we learn is changing. By allowing us to more easily share information and acquire knowledge, the Internet has made it easier to learn informally. Moving away from the traditional academic model, we are increasingly learning from each other and on the job.

Microsoft SharePoint’s popularity as a collaboration environment for everyday work tasks makes it a readily available environment for learning functions — an idea that fits in well with the 70+20+10 learning model. Assessments also fit in well with that model, and with SharePoint, too.

Many types of assessments can work well with SharePoint – everything from quizzes, diagnostic tests, knowledge checks and competency tests to surveys and course evaluations. No matter what the setting – a formal learning program, regulatory compliance, performance support or an employee/partner portal, perhaps – assessments have key roles to play.

How to include assessments in SharePoint?
•    Inbuilt SharePoint – functional for basic surveys
•    Custom web parts – write your own!
•    Embed Flash apps – possible for simple quizzes
•    Embed web apps – easy to do. (See how a Questionmark user has embedded a quiz to engage learners.)

If you would like to learn more about using assessments within SharePoint, you can check out this Questionmark presentation on SlideShare.

For more details, download the white paper Learning and Assessment on SharePoint or visit John Kleeman’s SharePoint and Assessment blog.

Steve Lay on Integrating SharePoint with External Systems

Posted by Julie Delazyn

Microsoft SharePoint’s popularity as a collaboration environment for everyday work tasks makes it a readily available environment for learning functions — an idea that fits in well with the 70+20+10 learning model.

Questionmark Integration Product Owner Steve Lay points out the possibilities and challenges of integrating learning functions into an existing portal environment in a recent post on John Kleeman’s SharePoint and Assessment blog.

In that post, Integrating with SharePoint: Intranet to Internet, Steve identifies issues such as integrating identity and authentication between SharePoint and external systems, notes the progress being made toward using externally hosted tools together with SharePoint and offers some links for people interested in more details.

The post is well worth reading if you are interested in SharePoint integration. For insights on many other integration-related themes, check out Steve’s own blog.

Looking back (and forward) to #qmcon

It’s three weeks since the Questionmark European Users Conference and I’m still following up on some of the conversations that started there. The presenters really help shape the theme of the conference: each session provides the starting point for the discussion that happens in the breaks, in the bars and on into our community spaces and discussion forums after the event.

An increasing part of this conversation is something often referred to as the back-channel. For me, the back-channel is a way that participants in an event can contribute to a wider conversation around the event’s themes using their mobile devices.  To help you grapple with the details of this concept, you might like to read this interesting blog post preparing participants for DevLearn 2011:  “What exactly is a back-channel?” I particularly like the comment from a speaker encouraging audience members to use their mobile devices to engage.

So how would you tune into the Questionmark European Users Conference back-channel? If you were using Twitter (and that is the easiest way) you would  just search for all tweets that include the special hash-tag #qmcon (‘hash’ is one of the many names for the # symbol on your keyboard).  Most Twitter monitoring tools make it easy to monitor your search continuously so that you can see new comments appearing in near real time.  To contribute to the conversation about this event, you would simply include the term #qmcon in your tweet.

Don’t be shy!  Of course, when you tweet publicly it might be recorded and preserved for all time (especially if you make particularly witty and incisive contributions) but back-channel comments are short and conversational and usually ephemeral like any other comment or question you ask in a discussion session. Why not give it go?  Next time you’re at a conference find out the back-channel hashtag and have a look.  If you’ve got a burning comment or question you don’t need to wait until the end of the session anymore: tweet it and see what comes back.

I always like to monitor the back-channel at every conference I attend, and when I’m presenting I always check back afterwards to see if there are comments and questions I need to follow up on or ways I can improve my session next time.  For example, here was a useful tip from someone in the audience of one of my sessions:

“#qmcon Acronym city in Integration session. developers have a language all of their own, no pun intended.”

Clearly I need to humanise that slide deck a bit for next time.

In my opinion, following hashtags is the way to use Twitter.  And it isn’t just something people do at conferences either.  Producers of live TV programmes monitor hashtags and provide feedback to the presenters in real time. You can also follow major trends and world events; try searching for something of interest and see what tags people are using to follow the conversation.

Still enjoying the Halloween spirit?  Just follow #pumpkin for a constant stream of pumpkin-related tweets and pictures.

Including a Questionmark Knowledge Check within SharePoint is easier than you think

Posted by John Kleeman

Many Questionmark customers use SharePoint within their organization. Microsoft SharePoint is a fantastic tool that lets non-technical people create collaborative web sites, and SharePoint is a great system to deploy assessments in for learning, training and compliance.

One of the easiest ways to include an assessment inside SharePoint is as a knowledge check – you can easily put a Questionmark Perception assessment beside some learning content as in the screenshot.

embed assessment sharepoint 2010

Putting a knowledge check in a SharePoint page gives three benefits

  • The learner can check he/she understands
  • The learner gets retrieval practice to reinforce the learning
  • As author, you can run reports to see which parts of the learning are understood or missed

In order to help people get the benefits of using assessments inside SharePoint, Questionmark have launched a new blog http://blog.sharepointlearn.com which focuses on SharePoint and assessment. This will allow us to run more detailed articles on SharePoint and assessments than the main blog can.

SharePoint is a lot easier to use than many people think. You don’t need administrative rights or programming skills to do most things. At the Questionmark Users Conference last week, I ran a session where people added an assessment in a sandbox site in just a few minutes. You can include an assessment inside SharePoint using the Page Viewer Web Part, which most people who can edit SharePoint pages have access to – if you want to give it a go, here are some instructions from the new blog.

Measuring the Impact of Social Learning

julie-smallPosted by Julie Delazyn

“The Social Network” – a movie about Facebook founder Mark Zuckerberg — was a big winner at the recent Golden Globe Awards in Los Angeles. And we’re pleased to say that social networks will be high on the agenda as Questionmark users gather in LA (with black ties nowhere in sight!) March 15 – 18 for the Questionmark 2011 Users Conference.

We’ll be paying considerable attention there to the growing importance of social networks as one aspect of a major shift in how people learn. Many organizations, recognizing that the vast majority of  what we learn comes from firsthand experience and from other people rather than from formal learning and study, are turning to informal and ‘social’ learning initiatives that use portals, blogs, wikis, forums and other tools. At this year’s conference we’ll be highlighting many new features and applications of Questionmark’s Open Assessment Platform that work in concert with many commonly used informal/social learning technologies – from embedded and observational assessment to new user interfaces and tools for content evaluation. (See a sneak peek below — and come to the conference for more on this!)

We’re looking forward to seeing you at the conference in March – we’re making sure the conference agenda will be packed with sessions and events that we’re sure you’ll be able to “like.”

Only two more days to register before the early-bird deadline!