Find the gaps, then design the learning

Posted by Jim Farrell

Having spent my entire career (amazingly coming up on 20 years now) in the learning industry, I find it curious that I often cannot find time for my own learning.  I always have a healthy stack of items on my nightstand that I have every intention of reading, but the stack is rarely diminished.

Twitter has become my digital nightstand. Instead of weeding through articles and websites, twitter has helped me “hire” my own personal curators to show me what is new and exciting in our industry. One recommendation from one of my curators, @cammybean, was the book, Design for How People Learn, by Julie Dirksen. Now the title alone is enough to grab most professional educators, but what really got me was @cammybean saying that most of the illustrations were stick figures. I am a huge fan of stick figures in illustrating learning so I was sold.

According to Julie, before you can create a good learning experience, you have to understand the gaps: knowledge, skills, motivation, environment and/or communication. I use and/or because it is likely that there are multiple gaps that must be addressed when dealing with more than one learner. How do we find gaps? We ask questions!

In my experience, simple surveys or assessments can help uncover most types of gaps. For example, letting participants rate the knowledge level on the topics that are to be covered can give a lot of insight about the class make-up. And if you really want to check their knowledge, ask them questions about things they say they have medium understanding of while avoiding topics about which they say they have minimal knowledge. Think about it: Why would you ask questions when the participant has already told you they don’t know the answer?

Asking participants how often they perform certain skills and how important they are can give you information about how much opportunity they will have to practice those skills back at work. It will also help you gauge how motivated they are to learn them. It’s also great to ask learners is if they know why they are attending the training. This can uncover communication gaps that could hinder their success.

We all know that evaluating participants’ progress throughout the learning process is valuable, but asking questions to determine the gaps participants are bringing to learning can make or break a training experience.

Looking back (and forward) to #qmcon

It’s three weeks since the Questionmark European Users Conference and I’m still following up on some of the conversations that started there. The presenters really help shape the theme of the conference: each session provides the starting point for the discussion that happens in the breaks, in the bars and on into our community spaces and discussion forums after the event.

An increasing part of this conversation is something often referred to as the back-channel. For me, the back-channel is a way that participants in an event can contribute to a wider conversation around the event’s themes using their mobile devices.  To help you grapple with the details of this concept, you might like to read this interesting blog post preparing participants for DevLearn 2011:  “What exactly is a back-channel?” I particularly like the comment from a speaker encouraging audience members to use their mobile devices to engage.

So how would you tune into the Questionmark European Users Conference back-channel? If you were using Twitter (and that is the easiest way) you would  just search for all tweets that include the special hash-tag #qmcon (‘hash’ is one of the many names for the # symbol on your keyboard).  Most Twitter monitoring tools make it easy to monitor your search continuously so that you can see new comments appearing in near real time.  To contribute to the conversation about this event, you would simply include the term #qmcon in your tweet.

Don’t be shy!  Of course, when you tweet publicly it might be recorded and preserved for all time (especially if you make particularly witty and incisive contributions) but back-channel comments are short and conversational and usually ephemeral like any other comment or question you ask in a discussion session. Why not give it go?  Next time you’re at a conference find out the back-channel hashtag and have a look.  If you’ve got a burning comment or question you don’t need to wait until the end of the session anymore: tweet it and see what comes back.

I always like to monitor the back-channel at every conference I attend, and when I’m presenting I always check back afterwards to see if there are comments and questions I need to follow up on or ways I can improve my session next time.  For example, here was a useful tip from someone in the audience of one of my sessions:

“#qmcon Acronym city in Integration session. developers have a language all of their own, no pun intended.”

Clearly I need to humanise that slide deck a bit for next time.

In my opinion, following hashtags is the way to use Twitter.  And it isn’t just something people do at conferences either.  Producers of live TV programmes monitor hashtags and provide feedback to the presenters in real time. You can also follow major trends and world events; try searching for something of interest and see what tags people are using to follow the conversation.

Still enjoying the Halloween spirit?  Just follow #pumpkin for a constant stream of pumpkin-related tweets and pictures.

My “#Follow Monday” Picks

jim_small Posted by Jim Farrell

I don’t know about the rest of you, but Twitter and Google+ have become my main ways of keeping up with industry happenings (as well as interesting celebrities).

The highlight of my week in terms of professional and personal growth is #FollowFriday. For those of you who are new to Twitter, every Friday, people list people they find interesting and think their followers should pay attention to. So, since today is Monday, consider today #FollowMonday Jim Farrell style with 10 people and companies I think are worth following.

@Questionmark – Did you really think I was going to start with something different? Following Questionmark gives you blog articles, videos and case studies for people interested in assessments.

@JaneBozarth – Jane is a fellow North Carolinian and the E-Learning Coordinator for the North Carolina Office of State Personal. That title is impressive but what is more impressive are her books, articles and blogging about learning and e-learning. A must follow!

@ambermac – Amber Mac was one of the keynote speakers and mLearnCon 2011 and is the host of a television show in Canada on emerging technologies. Her book, Power Friending, is a must-have for people in the world of communication and social media.

@Quinnovator – If you have been to an e-learning conference you know Clark Quinn. His sessions, blog articles and books on learning are always extremely thought provoking and lately seem to be focusing on mLearning.

@MegSecatore – Megan Secatore is an instructional technologist I follow most closely during e-learning conferences (although since she is in Boston — my hometown — I enjoy her non-work tweets too). She is an extremely active tweeter  — and she’s so good at it that I can learn all about the various conference sessions she attends by reading her tweets. She’s could outdo the play-by-play announcers at most sporting events!

@WillWorkLearn – Will Thalheimer is an old friend of Questionmark (and I mean friend for a long time, and not old). I would consider Will a learning visionary. Will is not a heavy tweeter but he is worth following to see which conferences he is attending.

@JohnKleeman – John is Questionmark’s founder and chairman. John is a very active tweeter and is currently passionate about Questionmark’s integrations with SAP and SharePoint.

@eLearningGuild – Besides having the second best conferences (behind Questionmark of course), the eLearning Guild is a tremendous source of information from e-learning, assessment and learning professionals. A definite must-follow.

@ASTD – Like eLearning Guild, the national and local ASTD chapters are a great way to learn about conferences and find interesting people to follow.

@marciamarcia – Marcia Conner was one of the first people I followed on Twitter. She has a great book titled The New Social Learning that I have read two times now. She is a speaker at many learning conferences and is someone to definitely check out.

“Follow Friday” — Some thought leaders to watch on Twitter

Posted by John Kleeman

I find Twitter a great way to learn every day and stay up to date with e-assessment and learning technology. With Twitter, I can see a summary of important information others think it worth me knowing.

If you’ve not tried Twitter and would like to get started, go to and sign up. Choose a short, memorable user name, set up your profile and choose some people to follow. You can then view their tweets in a browser or – more useful for many – on your mobile phone or iPad. And if you want to contribute, it’s easy to post your own tweets back.

Twitter has a “Follow Friday” tradition of recommending who to follow at the end of a week. Here are a few of the people I personally follow and suggest many of you might find interesting … I’’ll mention more in a future post!


Charles JenningsCharles Jennings used to be head of learning at Reuters and is now a freelance learning specialist. He’s one of the people who’ve popularized the 70+20+10 model for informal learning.

Example tweet: “An effective social media security strategy starts with user education”


drdjwalkerDavid Walker is Senior Learning Technologist at the University of Dundee, he’s also on the board of the E-Assessment Association and tweets on e-assessment.

Example tweet : Interesting article in this weeks @timeshighered about online exams and allowing students access to Internet/search tools. Worth a read.


Michael Feldstein

Michael Feldstein is the author of a longstanding educational technology blog and is a knowledgeable commentator on the academic market and their LMSs.

Example tweet: Hmm. The IMS has removed the word “standards” from the mission statement. #LI11


The Questionmark marketing team said I’d better put this one here :). No, seriously if you follow @questionmark on Twitter, you’ll get to hear of all our announcements and blog articles and can follow up on those you’re interested.

Example tweet: The fraud triangle: understanding and mitigating threats to test and exam security a slideshare presentation



Prashanth Padmanabhan is a product manager at SAP. He’s great at finding new ideas in business, software design and talent management and condensing them into a stream of tweets.

Example tweet : “How Do We Prepare Kids for Jobs We Can’t Imagine Yet? Teach Imagination”


Will Thalheimer

Will Thalheimer is one of the gurus of learning research, he reviews research from academics and applies them in a practical context.

Example tweet: Learning and Forgetting Curves — Implications Explained in New Video:

If you want to follow me on Twitter, you can find me as @johnkleeman. Check the tab at the left for Twitter addresses for several Questionmark colleagues – including our CEO @ericshepherd. I hope you find Twitter as useful as I do to learn every day.

Anticipating Questionmark user presentations at European conference

Posted by John Kleeman

Questionmark’s European user conference starts this Sunday in Amsterdam. I’m really looking forward to the event and wanted to share my enthusiasm with you.

Questionmark users are usually passionate about assessment and are often the visionaries and leaders of assessment within their organization. Questionmark staff are also passionate about assessment and work from many different places. So our  conferences – European and American – are a special opportunity for us all to get together in person, meet old friends, make new ones, and speak about assessment : how we can do it effectively on computers (and now mobile devices), and how to use and improve Questionmark software.

Usually my schedule at our conferences puts me in Product Central, engaging with focus groups on how the product should be better, but this year our product management team will be taking the lead there and I’m hoping to spend more time listening to our user experience presentations. This year in Europe we’ll hear the following case studies:

  • One of the world’s largest banks explaining how they are allowing people to invigilate (proctor) themselves but retain trust in the process for compliance purposes
  • Johnson Controls describing how they have implemented a certification programme with Questionmark
  • KU Leuven, one of the oldest universities in the world and with a great depth of technical knowledge, speaking on high-volume delivery and extending Questionmark
  • Experience from the Open University of the Netherlands about implementing Questionmark Perception in their freedom of time, place and pace philosophy
  • A session on advanced item types from an innovator at Vrije Universiteit Amsterdam
  • Experience from a new user, Rotterdam University, which started in 2009 but have already delivered 10,000 assessments
  • An update from ECABO about using Questionmark with psychometric validation for high stakes exams in the Netherlands

I hope to attend most of these sessions. And I’m looking forward greatly to seeing Questionmark colleagues, Questionmark partners and Questionmark customers, where for a few days we’ll exchange information, ideas and return to work re-charged to continue improving the world of assessment.

For those of our customers who are not coming, you can follow us on twitter (#qm10) and live blogging. We’ll also post handouts after the event. I hope you will join us in 2011 in Los Angeles or Europe.

Using Twitter to help learners retain knowledge

john_smallPosted by John KleemanTwitter  Logo

Here’s a question for you: “What is the best way of stopping people forgetting things after learning?”

Think about this for a moment before looking ahead if you can.

I hope your answer is something like this: by asking them questions over time after the learning takes place.

When you learn something, you connect two or more concepts in memory. And when you are asked a question about what you have learned, you have to search your memory to find the answer. This searching makes the connection in memory stronger, so in the future you will be more likely to remember what you have learned rather than forget it. If you’re not familiar with this important idea, see these white papers by learning expert Will Thalheimer for more information:  The Learning Benefits of Questions and Measuring Learning Results.

If your learners go on to another course or go back to work, it’s not always easy to reach them to stimulate their memory with follow-up questions. Here’s where Twitter comes in: it can be a great tool for sending follow up questions.

Twitter grad logo

  1. Have your learners follow you on Twitter, either on your main account, or on a subsidiary account made for each course.
  2. Post short questions as tweets to stimulate people’s memory. Remember, even thinking about the answer can help reinforce the learning. You could post the right answer the next day.
  3. Follow these up with quizzes in Questionmark Perception. You can post links to to these assessments in your tweets. With the new support of mobile devices in Perception version 5, your learners can access these quizzes from mobile devices as well as PCs and Macs, and take the quizzes from their home or while traveling.

Shortening a question into 140 characters  is usually possible, and it’s easy to compress a URL to Perception’s open access entry point (open.php) to fit within a tweet. For instance the URL links to one of Questionmark’s sample assessments on Electricity Skills.
I hope this idea helps. And in case you’ve forgotten, what is the best way of helping people remember after learning?