At a university with tens of thousands of students, how do you make it easy for students take assessments, and how do you manage the reams of data gathered from quizzes, tests and surveys? As manager of the Blended Learning Support Team at Glamorgan University in Wales, David Lewis thinks about these questions every day.

He’ll be sharing some of the answers during his case study presentation, Deploying and Using a Blackboard-integrated Questionmark Module Evaluation System, at the Questionmark European Users Conference in Brussels this fall.

David Lewis

I asked him recently about his presentation and what he’s looking forward to at the conference:

Tell me a little about the University of Glamorgan.

The University of Glamorgan is the second largest university in Wales, and we have nearly 24,000 students. Seventy percent are on campus but we also have a large constituency of part-time time students – about 6,000. We have a fairly even mix of traditional students and people who come back to university as adults. We are known all over the country for our technological innovations, particularly in the areas of assessment and simulations – and we will celebrate our 100th anniversary in 2013!

What kinds of assessments are you using there?

We use Questionmark for pretty much everything, but in some cases we also use other forms of assessment including “Turnitin” for plagiarism checking and submission of online material. We do a lot of in-class tests, exams, formative assessments and surveys. We run hundreds and hundreds of tests in engineering and accountancy – and in almost every subject area there’s some form of Questionmark assessment going on.

We gather a great deal of student intelligence, so surveys and questionnaires are very important to us. The surveys work in two ways –  giving us some information but also providing feedback to the students. For instance, we may ask students if they know which faculty they belong to. If they know, we’ll give them some details about their faculty, some resources, but if they don’t know we’ll tell them how they can find out and where to turn to for information and help. In this way, we can identify students who may be dropping through the net and help them get the information they need.

What will you cover during your case study at the Users Conference?

My presentation will be about how we use Questionmark Perception as our module evaluation tool. We use Blackboard Building Blocks, together with Perception, to make it easy for students to complete evaluations and for tutors and module managers to keep track of assessment data.

I’ll tell about our efforts to improve the availability of the tremendous amount of data we collect from assessments and to improve our survey response rates. I’ll explain our various iterations and improvements, including our delivery of multi-language assessments. And I’ll talk about how we’re trying to automate more administrative processes and make information more readily available to students.

I’ll show how we make as many of our systems as possible accessible to students via our custom portal. The portal gives us a nice baseline on which to integrate and provide all our functionality to students with a consistent look and feel. They can get course information on the portal, as well as their assessments. There’s one button to access assessments: they always know where to look for them! The portal works well for staff members, too: module leaders can click on a button to update student records. All the data we collect from evaluations and other assessments goes directly into Blackboard for the benefit of both students and academics.

Can you share some of the lessons you’ve learned?

It’s very important to make it easy for students to take evaluations. One way we do this is to give students a link that allows them to switch between English and Welsh version of a single questionnaire. That’s a really big advantage over having two separate questionnaires.

We’re also making it easy provide print-friendly versions of evaluations for instructors or students who prefer or need to use paper and pencil. This appears in a window within our Blackboard frame. This is our default for enabling academics to provide an alternative mode of evaluation.  But they can choose any alternative they want.

We’ve also found that it’s not always necessary to wait until the end of a module to run an evaluation. Some instructors like to run a mid-module evaluation. Or students might be notified about an evaluation two weeks before the end of the module, which gives them plenty of time to complete it.

We’ve organized our evaluations so that we can view results in different ways. By adding information about faculty, subject area, year of study and so forth, we can look at all first-year engineering modules, for instance, or third-year courses in the faculty of Health Science and get a wider view of what’s going on.

What are you looking forward to at the conference?

Number one on my list is anything to do with Questionmark Perception version 5! We will be upgrading very soon so I’ll make a beeline to any sessions touching on the upgrade process. I’d like to engage with the user community about their experiences and learn from them about that process. Also, it will be a nice opportunity for me to speak at the conference and engage with the company. And of course who could turn down the beautiful city of Brussels?

The conference will be held October 9 – 11. Click here to register.

With the program for the Questionmark European Users Conference filling up quickly, I got on the phone this week with one of the case study presenters, Onno Tomson.

Onno is a senior advisor at ECABO (Centre of Expertise on Vocational Education, Training and Labour Market for the economic/administrative, ICT and security professions.). Based in The Netherlands, ECABO runs a large number of tests and certifications for European organizations – everything from safety certifications for workers in the oil industry to  required tests for financial advisors.

Onno Tomson

Onno told me a little about ECABO’s longtime use of Questionmark and some of the new things they are doing. His presentation, Questionmark Can Be More than Questionmark, will explain how ECABO connects Questionmark Perception with third-party applications. It will also offer advice about how using the right tools, options and connections can improve efficiency.

Here’s our conversation in a nutshell:

Could you tell me how ECABO uses Questionmark Perception?

We create and deliver exams for four big organizations with participants in different countries. We are using Perception to create all the exams, so Questionmark is at the heart of our exam department. It’s our main software for creating all the content for several large item banks. Without Questionmark we would be doing this all by paper and pencil! Right now, we are helping one organization transition from paper-based tests to computerized tests. We use Questionmark for them too, using what we call “Questionmark to Paper “(Q2P). This is a custom-built application that uses Application Protocol Interfaces (APIs) to produce Questionmark Perception tests in PDF format and scan the results into Perception for reporting and analysis.

How many tests do you deliver each year, and in how many different languages?

We deliver tests in 12 different languages and still growing. We run 150,000  assessments a year just in the one organization running paper-based tests and transitioning to CBE (computer-based exams). In addition, we’re running 5,000 online assessments and in the end we expect to be running at least 100,000 online assessments per year. Paper-based assessments will always remain but the amount will decrease.

How do you optimize the efficiency and quality of your assessments?

We try to automate as much as possible all the steps in the process. For example, people register online and from that moment automatically a new group of participants is created and linked to a specific test center. And administrators of that test center automatically get direct access to new groups they are responsible for. The participants are also automatically scheduled for this assessment. QMWISe (Questionmark Web Integration Services environment) helps us make API connections with third-party applications, and we make the most of our Questionmark investment by using Questionmark Live, Translation Manager, Printing and Scanning, Result Warehousing and other options, plus some custom-built tools. These all help us with construction, distribution and reporting so that we can meet customer demands cost-effectively.

Could you tell me a little more about connections with third-party applications?

One of the main third-party apps is the app we created for producing paper-based exams. This application communicates using Questionmark APIs, and it can handle all kinds of variables and lay-outs, bubble sheets included. We also have a registration system that needs to talk to Questionmark, another app for sending records to our back office for invoicing, and one for connecting with a national database for certification candidates. All of these apps need to communicate with Questionmark, and we have learned a lot about using QMWISe to achieve this. I’ll be talking about that during my presentation.

How are you using Questionmark Live?

Questionmark Live is our major platform for our subject matter experts. As far as possible, construction of questions is going into Questionmark Live and from there into our production database.  We have between 100 and 200 SMEs. So being able to use Questionmark Live means that we are now getting more questions in the proper format for use in Perception instead of having to convert from documents in other formats. It’s a lot easier for us. And our SMEs are learning to use Questionmark Live for collaboration.

What are you looking forward to at the conference?

Meeting with the Questionmark staff, of course! And one of the main things I’m doing at the conference is going to Product Central and discussing how to equip Questionmark for the future. One of the things we would like to discuss is analyzing assessments using Item Response Theory.

The conference will take place in Brussels from the 9th to the 11th of October.

Check out the conference schedule and register soon.

I had a great time at the European Users Conference in Amsterdam.  Thanks to Stoas for their key role in making this wonderful event happen! Stoas are a learning consultancy based in Holland that provides Questionmark Perception-based solutions to education and business there.

As Questionmark’s integration products owner, I was especially interested to see  plenty of sessions that looked at integration issues, from customizing the templates used during assessment delivery right through to integrations with customer portals. I wish I’d had the opportunity during the conference to attend some of the best-practice sessions that were timetabled alongside my own. Fortunately, the conference has a dedicated space on our Community Spaces system and many of the presentations are appearing there so that I can catch up — thank you!

One session I did get an opportunity to go to was a session presented by Stoas themselves on their use of QMWISe (with a bit on templates). QMWISe is the name of our web service application programming interfaces (APIs). With QMWISe, system integrators can link assessment management into their other systems. It also allows programmers to create custom user interfaces to suit their own processes.  QMWISe is a key component of our open assessment platform.

I liked the way the presenters talked about how they distinguish between single sign-on and what they described as “single log-on”. Traditionally, single sign-on means a single challenge followed by access to multiple systems. For example, you might be prompted for your user name and password when you log in to your company portal and, from there, access many of your organization’s systems without having to identify yourself again.

With a common, weaker form of single sign-on, the same identity is used across multiple systems even though the user is challenged separately as they access each one. Stoas used the term single log-on when referring to the stronger requirement and demonstrated a system that used QMWISe to obtain a single log-on from a customized learning portal into Questionmark Perception. The presenters went on to show us an interesting dashboard view that used a blend of QMWISe and custom database queries where no suitable API exists (yet!).

The difference between sign-on methods can be quite subtle. I expanded on some of the common models of providing participant access in my own best-practice session. For workers or students with personal computers, a familiar pattern is a “remember me” checkbox. This causes the Web site to store the access credentials in the user’s Web browser as cookies, reducing the need for a single log on. (Windows Authentication on PCs works in a similar way.)

In the future, single-sign-on complexity seems likely to be handled directly by the system administrators who install and configure Web servers. Plug-in modules for web servers are now available that allow an organization to choose from a variety of different authentication systems (also known as “identity providers”) to protect the web applications they host.  For on-demand services, standard protocols are emerging that allow customers to link to their chosen identity providers without having to host the Web application at all.

Now I am looking forward to the U.S. 2011 Users Conference, where I hope to hear some more excellent presentations.

Questionmark have recently introduced multi-lingual apps to deliver assessments on Android, iPhone and iPad mobile devices, and I spent time at the European Users Conference last week demonstrating our Android app on my phone. In listening to our users talk about how they might deploy assessments on mobile devices, two areas attracted the most interest:

1. Formative assessments. Assessments help learning; so giving quizzes to people on the device that they carry around with them all the time is interesting to many, and is easy to do. It’s attractive that with Questionmark, you author the assessments once, and participants can take them on their device of choice.
2. Observational assessments. There is a big need to assess people doing things in work situations, whether it’s how a welder uses a machine, how a nurse interviews a patient or how a soldier cleans his equipment. It’s not practical to carry a laptop into many work environments, but it is practical to fill in questions on a mobile device while observing someone, so this possibility attracted a lot of interest.

Questionmark will shortly be improving our software for use for observational assessments – so watch this space.

P.S. I’m very excited to hear that Bryan Chapman is going to be the keynote speaker for our next users conference in Los Angeles in March 15 – 18, 2011. He is a real thought leader in our industry and someone I admire greatly – he introduced me and Questionmark to the AICC standard back in the 1990s and encouraged us to support it  -  and we’ve now been re-certified 5 times!

Bryan Chapman

With the Questionmark European Users Conference just completed in Amsterdam this week, we are gearing up for the 2011 Users Conference in Los Angeles March 15 – 18.

We’re very pleased that Bryan Chapman will present the keynote address there. As Chief Learning Strategist at Chapman Alliance, Bryan is involved in providing research-centric consulting solutions that help organizations define, operate and optimize their strategic learning initiatives. His talk on Assessment’s Strategic Role in Enterprise Learning: Innovation and Trends will include examples of innovative assessment strategies that have improved learning and performance within many different organizations.

This video will give you a general overview of what happens during the conference.

Questionmark’s European user conference starts this Sunday in Amsterdam. I’m really looking forward to the event and wanted to share my enthusiasm with you.

Questionmark users are usually passionate about assessment and are often the visionaries and leaders of assessment within their organization. Questionmark staff are also passionate about assessment and work from many different places. So our  conferences – European and American – are a special opportunity for us all to get together in person, meet old friends, make new ones, and speak about assessment : how we can do it effectively on computers (and now mobile devices), and how to use and improve Questionmark software.

Usually my schedule at our conferences puts me in Product Central, engaging with focus groups on how the product should be better, but this year our product management team will be taking the lead there and I’m hoping to spend more time listening to our user experience presentations. This year in Europe we’ll hear the following case studies:

• One of the world’s largest banks explaining how they are allowing people to invigilate (proctor) themselves but retain trust in the process for compliance purposes
• Johnson Controls describing how they have implemented a certification programme with Questionmark
• KU Leuven, one of the oldest universities in the world and with a great depth of technical knowledge, speaking on high-volume delivery and extending Questionmark
• Experience from the Open University of the Netherlands about implementing Questionmark Perception in their freedom of time, place and pace philosophy
• A session on advanced item types from an innovator at Vrije Universiteit Amsterdam
• Experience from a new user, Rotterdam University, which started in 2009 but have already delivered 10,000 assessments
• An update from ECABO about using Questionmark with psychometric validation for high stakes exams in the Netherlands

I hope to attend most of these sessions. And I’m looking forward greatly to seeing Questionmark colleagues, Questionmark partners and Questionmark customers, where for a few days we’ll exchange information, ideas and return to work re-charged to continue improving the world of assessment.

For those of our customers who are not coming, you can follow us on twitter (#qm10) and live blogging. We’ll also post handouts after the event. I hope you will join us in 2011 in Los Angeles or Europe.