Role-Based Permissions: A How-To Guide (Part 1)

Bart Hendrickx SmallPosted by Bart Hendrickx

If you manage which users can access your Questionmark environment and define what they can do when they log on, you know that controlling access can take time.

I’d like to take you through a two-part scenario that will demonstrate how to control access efficiently and navigate changes in job roles effectively.

Interested in learning more about role-based security in Questionmark OnDemand? I will be presenting a session on effectively managing users at the 2016 Questionmark Conference in Miami.

Imagine welcoming Ella to your team as a new colleague. She will be using Questionmark, and you want to make sure she has access to those functions of the software she will need—no fewer, and certainly no more. Then you ask yourself: What will Ella be doing? What will be her role?ella

Maybe the answer is along the lines of “she will be replacing Wendy.” Then you might wonder, “OK, so what was Wendy’s role?”

It is natural to think about what people do and the roles they play when you discuss what they should be allowed to do in a software system. And you may be accustomed to using Questionmark Enterprise Manager’s profiles to set things up, by storing a set of permissions in a profile, creating an administrator account and linking the profile to it.

This works well until you come across a colleague who will be actually doing more than one thing (don’t we all?).

“Yes, Ella will be replacing Wendy, but she will also be taking some of Bill’s workload.” You have assigned Wendy’s profile to Ella; she can now run several reports. You want to assign Bill’s profile to her as well. You can’t assign Bill’s entire profile to Ella, but you can compare the two profiles to see where they overlap. It turns out that in addition to what Ella has inherited from Wendy’s profile, Bill can run all reports, so you add the permissions for the remaining reports directly to Ella’s user account.

Two months later, your team restructures some of its operations. “We won’t be running Grade Book reports anymore and we want that permission removed from the users.” You think hard. Who was it that had this permissions? Can’t I just edit the profiles? That won’t update existing users. And what about those users who had the permissions applied directly to them? I can’t judge that merely by the profile that is attached to them.  I’d better edit all those users one by one to be sure.

You stare at the list of almost 50 administrators in your system, decide to get a coffee, sit down and take a breath. This is going to take a while.

In my next article, I will explain how to avoid this pitfall by setting up permissions more efficiently.

Interested in learning more about Role-Based Security in Questionmark OnDemand? I will be presenting a session on effectively managing users at the 2016 Questionmark Conference in Miami, April 12-15. Register by March 3 for a final chance to take advantage of our early-bird discounts…click here to register and learn more about this important learning event. Hope to see you in Miami!


Online Proctoring – An Invasion of Privacy?

Steve Lay Headshot

Posted by Steve Lay

Many organisations looking to expand their online offerings now use a new method to securely deliver high-stakes exams online: Online proctoring. A live proctor uses your computer’s webcam to observe you taking the test, to ensure its integrity. To make sure you work alone, the proctor asks you to scan your webcam around the room you are in. The proctors also asks you to show photo ID to verify your identity and will use screen-sharing technology to view your computer screen. In addition, secure browser software can sometimes be used to restrict other computer applications (such as opening a web browser) to restrict a test-taker from accessing digital resources.

Being watched in this way during an online exam often poses questions about privacy…

Is online proctoring an invasion of privacy? Do proctors still have access to your computer after the exam is complete? What sort of things can they access while you’re taking the exam? Can they access your files and identifiable information?

A video link with an online proctor invades no more privacy than taking an exam at a traditional face-to-face test centre. In many cases, allowing a proctor to see everything on your computer screen is just like a proctor at a test centre who can look over your shoulder, see your computer screen and prevent any restricted behavior.  But some online proctoring systems go even further, providing proctors with full control over a candidate’s computer.

Having a proctoring service take control of a candidate’s computer can often be quite helpful.  For instance a proctor who is trained in diagnosing and correcting setup issues can help speed up a process and can quickly resolve problems with the video or audio on the computer. A proctor can also guide the candidate through the exam software, in some cases entering special purpose access credentials that have not previously been made available to the candidate.

Although screen sharing and remote control solutions can be used with Questionmark Online Proctoring, there are alternatives for situations in which such far-reaching access to the candidate’s laptop is inappropriate. Using Questionmark Secure in conjunction with Questionmark OnDemand supports a special mode for online proctoring that gives the proctor limited proxy controls instead of complete control over the machine. For example the proctor can manage the running of the assessment without having control over the participant’s machine. The sense of ‘control’ that many proctoring solutions require here is  similar to popular screen sharing systems that allow you to “Give Control” or “Request Control”. Questionmark Online Proctoring does not require this, because the proctor is connected directly to Questionmark’s service and can manage the exam without going ‘through’ the participant’s computer.

In addition to the privacy advantages of these proxy controls for the candidate, this arrangement also enables the test content to be kept hidden from the proctor. This could provide advantages to the test provider over and above what can be achieved even in a test centre.  The proxy controls allow the proctor to pause the test, add extra time and even terminate the test completely. Meanwhile, Questionmark Secure takes care of monitoring the local computer for signs of misuse and flagging or preventing attempts to cheat.  Questionmark Secure can be audited and installed by a trusted system administrator for a company-owned laptop without having to provide the same permissions to the end user.  Questionmark Secure does not install keylogging software, or any other persistently active service.  It is only active during the exam process itself.

Interested in learning more about Online Proctoring? I will be presenting a session on ensuring exam integrity with online proctoring at Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15.

There’s only one day left to take advantage our earl-bird savings…click here to register and learn more about this important learning event. See you in Miami!

How many questions should you have in a web survey?

John Kleeman Headshot

Posted by John Kleeman

Web surveys offer a quick, effective means of gathering data and attitudes that can help you make decisions and improvements. But how many questions should you ask? What is the best length for a web survey? Here are some tips:

Want to learn more about survey techniques? I will be presenting a session on harnessing the power of your surveys at the 2016 Questionmark Conference in Miami April 12-15.

Research evidence

The best survey length depends on the survey purpose and audience, but here are some useful research findings:

  • The market research industry has studied ideal survey length in detail. In such surveys participants are often panel members or people with time who can be motivated or incentivized to answer longish surveys. A debated but often quoted rule of thumb in market research is that 20 minutes is about as long as a typical person can concentrate on a survey and so surveys should be no longer than 20 minutes.
  • In typical web surveys, dropout rates increase with a larger number of questions. For example one controlled study found  a drop-out rate of 29 percent on a 42-question web survey compared to a smaller dropout rate 23 percent on a 20-question one.
  • In long web surveys, participants often reduce time spent answering later questions, which can mean less accurate answers. This is an example of satisficing – participants not thinking too hard about how to answer but just giving an answer. Survey Monkey did an analysis of 100,000 real-world web surveys and found that for surveys of 3 – 10 questions, participants spent an average of 30 seconds answering each question, whereas for surveys of 26 – 30 questions, participants spent an average of 19 seconds.  So a longer survey may get lower-quality answers.
  • Task difficulty also matters. Shorter isn’t always better. Research (for example here) identifies that difficulty matters as well as length. Participants may abandon a survey when faced with too hard questions, when they would be willing to fill in a longer, less challenging survey.
  • Mobile users often have a reduced attention span, and it can take longer to answer questions on a smartphone than on a PC. One experienced commentator suggests that surveys take 20 – 30 percent longer on a mobile device.

So how long should your survey be?

There is no single right answer to this question, here are some tips:

Editing a jump block - choosing to skip to end of assessment if previous question was not applicable1. A key factor is the engagement of your participants. You can risk a longer survey if your participants are motivated. For example participants who have just undergone a three day course will be more motivated to fill in a longer survey about it than someone who’s just done a short e-learning session.

2. Consider using  branching to skip any unneeded questions.

3. Ask concise questions without lengthy explanations, this will reduce the apparent length of the survey.

4. Pretest your survey to try to remove difficult or confusing questions – a longer, clearer survey is better than a shorter, confusing one.

5. If your survey covers very different topics, consider breaking it down into two or more shorter surveys.

6. Make sure results for each question are actionable. There is no point asking questions where you aren’t going to take action depending on what you discover. Participants may disengage if their answers don’t seem likely to be useful .

7. Look at each question and check you really need it. As your survey length increases, your response rate will drop and the quality of the answers may reduce.  Work out for each question, whether you need the data badly enough to live with the drop in quality. Ask as few questions as you need – some successful surveys (e.g. Net Promoter Score ) just ask one question. Very often an effective and actionable survey can be ten questions or less.

Want to learn more about survey techniques? I will be presenting a session on harnessing the power of your surveys at the 2016 Questionmark Conference in Miami April 12-15. There’s only 1 week left to take advantage of our early-bird discount. Sign up before January 21 and save $200! I look forward to seeing you there!

Read more >

Hurry! Early-Bird Registration Ends Soon!

Postcard BannerThere is just one week left to save on your registration for the 2016 Questionmark Conference… And seats are going fast!

Take advantage of the early-bird discount and save $200, if you sign up by Thursday, January 21!

Aside from some fantastic social events that are designed to assist you in networking with other assessment professionals, we have a jam packed agenda covering topics such as:palm tree emoji 2

Bring your colleagues! Group discounts are available. Save up to $200 for additional registrants.

So meet us in sunny Miami April 12-15 and leap into the future of assessment and authoring!

Book your room at the Marriott Biscayne Bay at a discounted rate, while they are still available. Rates expire March 21, 2016.

Certification in the Cloud and the Move to Online Proctoring: An interview with SAP’s manager of global certification

John Kleeman Headshot

Posted by John Kleeman

I recently interviewed Ralf Kirchgaessner, SAP’s manager of global certification, about how the cloud is changing SAP certification. This is a shortened version of my conversation with Ralf. To read the full previously published post, check out this SAP blog.

John: What are the key reasons why SAP has a certification program?

Ralf: The overall mission of the program is that every SAP solution should be implemented and supported ideally by a certified SAP resource. This is to ensure that implementation projects go well for customers, and to increase customer productivity while reducing their operating costs. Customers value certification. In a survey of SAP User Group customers in Germany and the US, 80 percent responded that it was very important to have their employees certified and over 60 percent responded that certification was one of the criteria used to select external consultants for implementation projects.

John: What important trends do you see in high tech and IT certification?

Ralf: What comes first to the mind is the move to the cloud. Throughout the technology industry, the cloud drives flexibility and making everything available on demand. One aspect of this is that release cycles are getting quicker and quicker.

For certification, this means that consultants and others have to show that they are always up to date and are certified on the latest release. It’s not enough to become certified once in your lifetime: you have to continually learn and stay up to date. But of course if you are taking certification exams more often, certification costs have to be much lower. In some regions, people have to travel large distances to get to a test centre. With more frequent certification, it’s not practical to travel to a testing centre every time you take a certification. So our aim is to allow certification anytime and anywhere using the cloud.

John: How does online proctoring work for the candidate?

Ralf: A remote proctor monitors the candidate via a webcam, and there are a lot of security checks done by the proctor and by the system. For example, a secure browser is used, the candidate has to do a 360 degree check of his or her room, and there are lots of specific controls. For instance, you aren’t allowed to read the questions silently with your lips in case someone is watching or listening.

The great advantage to the candidate is flexibility. If someone says, “I’d like to do my exam in the middle of the night or on weekends because during the week I’m so busy with my project,” they can. They might say that they’d like to do their exam on Saturday afternoon: “After spending two hours playing with my kids, I’m relaxed to do my exam!” It’s such a flexible way to get certified and to quickly demonstrate that they have up-to-date knowledge and are allowed to provision customer systems.

John: Who benefits from certification in the cloud? Candidates, customers, partners or SAP?

Ralf: Of course, I think all benefit! Candidates have flexibility and lower cost. Customers can be sure that partner consultants who work for them are enabled and up to date. For partners, it’s a competitive advantage to show that their consultants are up to date, especially for new technologies like S/4HANA and Simple Finance. A partner is much more likely to be chosen to deploy new technologies if they can demonstrate that they have several consultants already certified in something that’s just been released. And for SAP, our goal is to have engaged consultants, happy partners and lower support costs. So everyone genuinely benefits.

John: What are some of the challenges?

Ralf: One example is that it’s important in cloud certification to get data protection right. SAP have very detailed requirements that we ensure our vendors like Questionmark meet.

Security is also a challenge. You need to prevent cheating and stealing questions.  And interfaces and integration need to be right. We have worked out how we get the data from our HR systems, how people book and subscribe to exams and then how they can authenticate with single sign-on into the certification hub to take cloud exams.

The delta concept also gives challenges. You need very precise pre-requisite management logic, where the certification software checks for example that, if you want to take the delta exam, you have already passed the core exam. It also can sometimes be difficult to prepare a good delta exam, particularly if a new release has very specific or detailed features, including some that apply in only some industries.

Lastly, providing seamless support is a challenge when using multiple vendors. The candidate doesn’t care where a problem happened: he or she just wants it fixed.

John: Where do you see the long term future of high-tech certification? Will there still be test centres, or will all certification be done via the cloud?

Ralf: Test centres won’t disappear at once, but there is a trend of moving from classroom-based learning and testing to learning and certification in the cloud. The future will belong to anytime, anywhere testing. The trend is for test centre use to decline, but it won’t happen overnight!

John: If another organization is thinking of moving towards certification in the cloud, what advice would you give them?

Ralf: Ensure that you are aware of the challenges I mentioned and can deal with them. And do some pilots before you try to scale.

Interested in learning more about Online Proctoring? I will be presenting a session on ensuring exam integrity with online proctoring at Questionmark Conference 2016: Shaping the Future of Assessment in Miami, April 12-15. I look forward to seeing you there! Click here to register and learn more about this important learning event.