Conference Close-up: Silke Fleischer on Design Strategies for Mobile Devices

Posted by Joan Phaup

Mobile assessment is high on the list of topics we’ll be discussing at the Questionmark Users Conference March 15 -18. Silke Fleischer, a co-founder of ATIV Software, will show attendees how to create materials that work well on mobile devices and manage their limitations during her talk on Getting the Most from a Small Screen: Design strategies for mobile devices.

Silke Fleischer

Silke has been a featured speaker at several national and international conferences of organizations such as the eLearning Guild and ASTD, and we’re very pleased she’ll be joining us in Los Angeles. I spoke with her recently about her work and her plans for presenting at the conference:

Q:   Could you tell me about your work at ATIV software?
A: We develop mobile applications for events and eLearning. Our core EventPilot application is a support system for conferences, events and meetings, and we are working on a mobile course app that helps training departments reduce printing costs for instructor-led training. We also do custom mobile development. In case you are wondering about how we got our company name, it stands for innovATIVe, creATIVe, alternATIVe!

Q: What trends are you seeing in mobile learning and assessment?
A: There are more than 700,000 smartphones sold per day, and learning is very convenient on them. More and more standard eLearning content is now being made available and optimized for mobile devices, especially with the technology getting better and screen resolutions larger. That’s not to say that classroom training is going away! You have to have the face-to-face interaction. I think we will see a marriage between mobile and the classroom.

One interesting trend is the use of mobile delivery within Instructor-Led Training (ILT) to cut printing and reduce budgets for handouts and course materials. Our upcoming course app, for instance, allows learners to log in and download all course-related PowerPoint presentations on an iPad and take notes directly on the device to a particular slide. When the course is over, they can take their assessment using Questionmark’s iPad App and also send themselves an email with notes and marked items for easy reference about what they have learned. Also, you can update content on the fly and integrate social aspects such as your training blog or company messaging system. This approach can save thousands of dollars in printing costs for large organizations over the year.

Q: What are the key elements that need to be considered in designing and implementing mobile assessment?
A: Mobile devices are unlike desktop computers or laptops. The smaller screen, for example, limits some types of training and assessments. You also have to consider software and hardware. On the software side, you might have to find new ways of creating interactive content if there is no Flash player available  When it comes to hardware, even though devices are very fast these days, their performance is still dependent on the processors and on memory limitations, so you will want to invest in good programmers. Wifi and cell reception requirements should be analyzed, and you can also look at native versus hybrid versus web apps. The amazing new capabilities outweigh the limitations. Today’s mobile devices make new training paradigms possible and enable instructional designers to use motion sensing and location-based services, cameras and advanced touch screens.

Q: What will you be sharing during your presentation at the Questionmark Users Conference?
A: Since mobile devices have such a small screens we will be talking about design and development approaches. I’ll be offering some basic guidelines to help attendees get started with small screen design and content presentation. An important aspect of this will be the fact that the screens aren’t only used for displaying content, but also for navigation with your fingers. I’ll be sharing some workarounds that will help people overcome the challenges presented by small screens.

Q: What are you looking forward to at the conference?
A: I’m eager to learn about other trends in learning and assessments, which I think will help me get inspired with new ideas for 2011. And of course it will be great to network with others  and share ideas. I’m really excited about going to the conference!

We’re looking forward to the conference, too, and hope you will check out the program and register soon!

Mobile Delivery: Using Questionmark’s App for Apple® iPhone® and iPad™

Posted by Julie Delazyn

Questionmark’s auto-sensing delivery makes it easy to author an assessment once, and then deliver to many different types of devices. The same Questionmark assessment you deliver to desktop or laptop can also be delivered via standard browsers on many different smartphones and tablets – but today I want to highlight the Questionmark Apps for Apple® iPhone® and iPad™. Plainly put, the Questionmark App equals added convenience. With simple configuration and options for customization it offers organizations an easy way to deliver quizzes and surveys to learners on the move, giving participants one-touch access to assessments that have been assigned to them.

To get the Questionmark App:

  1. Visit the App store and search for “Questionmark”
  2. Choose either Questionmark’s iPad App or iPhone/iPod touch App
  3. Install the App on your device

Once installed:

  1. Try a Demo and see examples of different assessments for you to try
  2. Configure the app to point it directly to your Perception Server. You only have to configure the application once; it will remember the settings.

To easily configure the App, customers can enter the URL of their Questionmark Perception server or enter a customer number provided by our Customer Care Team.

Check out this video to see how to configure the app:

For more on mobile delivery be sure to attend the Questionmark Users Conference in Los Angeles, March 15-18. View the conference program and register soon!

Conference Close-up: A Tiered Assessment Model for Internal Certification

Posted by Joan Phaup

One of the things I most look forward to at the Questionmark Users Conference is the opportunity to hear customers’ case studies.  Accenture Certification Specialist Neetu Jaiswal spent a few minutes telling me about one of her session about a Tiered Assessment Model for Internal Certification, that she will co-present with Accenture Certifications Analyst Supriya Shetty.

Neetu Jaiswal

Q: Tell me a little about your company.

A:  Accenture is a global management consulting, technology services and outsourcing company. We have about 204,000 employees serving clients in more than 120 countries. Our clients turn to us for industry and business process expertise to help them improve their performance. Our Business Process Outsourcing business involves 33,000 people serving more than 150 clients around the world.

Q: What’s your role in Accenture?

A: As a Certification Specialist, I focus on our certification program for Accenture employees which is organized in three tiers or levels, which build on assessment complexity and rigor. I am in charge of all the Tier 1 or Curriculum Level assessments that we offer in Accenture for various industries groups. I drive all activities associated with curricula and assessment design and development, including job task analysis and/or item writing workshops. I work with solution management and functional architecture teams to develop, deploy and support a cost effective, sustainable solution for the clients. I also oversee the administrative functions including help desk/e-mail support, reporting, and other functions as defined and agreed upon with the client.

Q: How are you using Questionmark?

A: We use almost all the features of Questionmark to support the three-tiered assessment model for Accenture’s internal certification efforts. A few years ago, we started with technical Certification exams in Questionmark and now we use Questionmark tools for technical certifications and other industry assessments, surveys, tests, scoring rubrics, end of program evaluations, etc.   We leverage several Questionmark tools – including Questionmark Live and the Results Management System – to support the development process.

Q: What you will be talking about during your conference presentation?

A:  The demand for employee testing and certification across Accenture has increased dramatically in the past two years.  Accenture’s certification teams had no choice but to look for new, innovative ways to develop tests in less time and with less money than in years past, while still maintaining a sufficient level of quality.

This presentation will describe the three-tiered assessment model used to support Accenture’s internal certification efforts. The three types of assessments are distinguished by the nature of the knowledge and skills being assessed, the types of questions used, the number of forms of each assessment, and the extent of psychometric analysis conducted throughout the development process.   Through the session, we will highlight how several Questionmark tools – including Questionmark Live and the Results Management System – are leveraged to support the development process.

Q: What are you looking forward to at the conference?
A: I want to find out what Perception version 5 offers and how we can use it to our benefit.  Also, I saw that there are many good papers to be presented, so I’m looking forward to attending those sessions. And I want to learn more detail about the psychometric analysis. Questionmark puts on a great conference with many networking opportunities, so I’m looking forward to being with other Perception users to learn about various best practices that they follow. I am also looking forward to meeting the Questionmark people with whom we interact but rarely meet face to face.

Indeed, the conference is a great place to meet face to face with Questionmark staff, industry experts and other Questionmark users. Sign up soon and join us in Los Angeles March 15 -18.

How SAP ensure their questions are good quality

Posted by John Kleeman

Questionmark are partners of SAP, and as well as blogging on the Questionmark site, I also blog on the SAP Community Site.

Sue Martin

I’ve recently started a series of Q&A interviews on thought leaders in the field of assessment and learning within SAP, and last week I interviewed Sue Martin, who is Global Certification Manager at SAP. I thought Questionmark blog readers might be interested in one question I asked her about how SAP ensure their questions are good quality.

John : What effort goes in at SAP to make the questions good quality and set the scoring fairly?

Sue : We follow industry standard practices to get legally defensible exams into place.  In order to get more experiential subject matter expertise, we moved from instructors writing questions to much more involvement from our own consulting organization. At the same time, to get the level of consistency and quality of review, we significantly increased the number of subject matter experts (SMEs) involved.

Here is an overview of the steps we follow:

  1. The first step is job task analysis to identify the tasks that we ask questions on. These are validated with customer organizations, partners and with our own consulting organization.
  2. The next step is the design phase, where we have the item writing workshops that write and review the questions. It depends on the subject matter, but there are normally at least 6 or 7 SMEs that sit in these workshops, supported by a psychometrician.
  3. We then have peer review from another set of SMEs after the workshop. By SMEs I mean primarily people who are actually working in that area, so for a Professional certification that would be a Senior consultant or a Platinum consultant.
  4. Then another group of SMEs go through the exam as beta testers and we look at the results.
  5. Then a group of experts goes through the questions and do a pre-standard setting process, estimating the difficulty level of each question.
  6. Then we have the final standard setting meeting to get to the cut score (pass score) determination phase.  And then at the end of all that, we finally have the exam!

To see the rest of the interview, including questions on how SAP are finding Questionmark effective for authoring and delivery of exams and how they are transforming their certification programme, click here.

Conference Close-up: John Kleeman on Using Research Findings to Improve Assessments

Joan Phaup

Posted by Joan Phaup

Current psychological research confirms that answering questions helps people remember what they’ve learned. But how can this research be applied to creating effective quizzes and tests?  Questionmark Chairman John Kleeman will address that question during a presentation at the Questionmark Users Conference on Using the Latest Learning Research to Improve Your Questionmark Assessments.

John has taken a keen interest in this research, which he has written about it in a previous post, and an article in yesterday’s New York Times shows that research in this area is getting mainstream attention.

John Kleeman

I thought it would do John some good to answer some questions about what he’s been learning and what he plans to share with his audience!

Q: What research findings do you regard as most relevant to the use of assessments in learning?

A: There’s a huge amount of psychological research out there, but what I still find stunning — although the basics have been known for a while — is that if you study something and don’t answer questions about it you are likely to forget it over time, but if you do answer questions it gives you retrieval practice and you are more likely to retain it. So the old-style practice of teaching people something and then quizzing them on it really does work.  People who’ve been away from school think, “Oh, I don’t need to take quizzes and tests anymore,” but they really should take quizzes and tests because the process of answering a question and getting retrieval practice really helps you learn.  This is an example of a desirable difficulty – something you don’t want to do, but which helps you learn.

Q: Why is retrieval practice so useful?

A: As I understand it, when you answer a question it gives your brain practice in retrieval and makes the pathway from that information to how you can apply it stronger…in layman terms, it makes those brain pathways stronger. The evidence shows that if you do answer things after learning you remember it better. If you want to learn something and retain it, be sure to answer some questions about it. Even if you answer a question only to yourself you are improving your retention.

Q: Is this an academic session?

A: Absolutely not! What I’m looking to do is to highlight some of the research that has been done — including new research from the last few years — and to suggest practical things that people using Questionmark software can to do take advantage of it.  I’ll be covering the retrieval effect and also how pre-questions can be effective and on how to use feedback effectively in Questionmark Perception. The session is  about how we can practically use the results of research.

Q: Can you tell me about some of the interactive exercises you will be doing during this breakout session?

A: I’ve got a few up my sleeve! One is that I plan to divide the room into two random groups. One part will study something on its own, without answering any follow-up questions; the other will study and answer some questions. A week afterwards I’ll run a voluntary assessment with those two groups and see which group performs better. I want to demonstrate that questions really do help you learn.

Q: What will people get out of the session?

I went to a training course recently. There were about 20 people, and at the end of the course there was an optional test. Only about six people took it. The others probably thought they had better things to do with their time or might have been concerned about their scores! But I want my audience to understand how much value there is in assessment  and how it can help people be effective learners. We’re missing something here and there’s a great opportunity for us to do better. I want to empower attendees with insight so that they can go back to their organizations and be more effective.

Today’s the last day for early-bird  registration for the conference in Los Angeles March 15 – 18. So I hope you will view the conference program and register soon!

Measuring the Impact of Social Learning

julie-smallPosted by Julie Delazyn

“The Social Network” – a movie about Facebook founder Mark Zuckerberg — was a big winner at the recent Golden Globe Awards in Los Angeles. And we’re pleased to say that social networks will be high on the agenda as Questionmark users gather in LA (with black ties nowhere in sight!) March 15 – 18 for the Questionmark 2011 Users Conference.

We’ll be paying considerable attention there to the growing importance of social networks as one aspect of a major shift in how people learn. Many organizations, recognizing that the vast majority of  what we learn comes from firsthand experience and from other people rather than from formal learning and study, are turning to informal and ‘social’ learning initiatives that use portals, blogs, wikis, forums and other tools. At this year’s conference we’ll be highlighting many new features and applications of Questionmark’s Open Assessment Platform that work in concert with many commonly used informal/social learning technologies – from embedded and observational assessment to new user interfaces and tools for content evaluation. (See a sneak peek below — and come to the conference for more on this!)

We’re looking forward to seeing you at the conference in March – we’re making sure the conference agenda will be packed with sessions and events that we’re sure you’ll be able to “like.”

Only two more days to register before the early-bird deadline!