13 Scary Questions to Ask your Assessment Cloud Provider

Posted by John Kleeman

As its Halloween I thought you might enjoy learning about 13 questions that might scare your Assessment Cloud provider.

Let me first share some background information …13 Scary Questions to Ask your Assessment Cloud Provider

With increasing use of Cloud systems like Google Docs, Microsoft’s Office 365, and Amazon, and with enterprise software giants like Oracle and SAP offering OnDemand services, many organizations that previously managed IT internally are delegating the running of servers. A Cloud service can save you money, and allow you to focus on core business and user issues, by letting someone else deal with the technology.

Secure and scalable Assessment Clouds are the next wave of tools available that help organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and successful learning outcomes
As you consider moving your assessments to a Cloud, you need to ensure your provider is offering the best possible service, security and data protection. You want a provider who is fully invested in giving strong security, scalability, elasticity and robustness, not just someone running a server under a desk! Exam security has different challenges and demands to other kinds of IT due to the confidentiality of personally identifiable information, questions and results, so you need to make sure that the system you use is safe and secure.

Here are 13 questions you might scare the less professional Assessment Cloud providers in the marketplace:

1. Do you host assessments in a well-established Data Center, certified to SAS 70 Type II, SSAE 16 Type II or ISO 27001?

2. Does your Data Center have multiple connections to the power grid with onsite generators with at least 24 hours fuel onsite in case of power outages?

3. Does your Data Center have multiple, fast Internet links so that if one goes down, connectivity remains?

4. Is every server in the system load balanced and does every component have redundancy, so that if any one system fails, another can take over?

5. Is browser access to assessments and administration protected by SSL (or TLS) to 128 bits or higher, so that assessment data and results cannot be intercepted on the Internet?

6. Do you follow industry good practice in software development to reduce surface areas of attack and protect against security vulnerabilities? Common methodologies to work with are called STRIDE and DREAD.

7. Do you have separate development/integration areas and staging areas to test on before deploying to production?

Questionmark’s OnDemand Testing and Deployment Process
Questionmark’s OnDemand Testing and Deployment Process

8. Do you have a data security policy for your employees who run the service to ensure that they maintain the secrecy of customer data? Does the policy include confidentiality agreements, background checks on employees, regular training, and regular testing of employees to check they that understand data security?

9. Can I see information on real time information on the current status and uptime, and access statistics from round the world? See status.questionmark.com for an example of what you might look for from a provider.

10. Is the service monitored and run 24/7 at both Data Center, network, hardware and application level, so that problems out of hours will be fixed?

11. Are results data backed up safely at least once an hour, so that in the event of a catastrophe, you should never lose more than an hour’s worth of data?

12. What access might government agencies have to data of foreign nationals and are your systems Safe Harbour Certified?

13. What is your track record do you have for being a trustworthy provider with references and case studies to back your claims up?

The answers to these questions for Questionmark’s OnDemand Service are all yes. If you want to find out more, read more details in our new white paper, Security of Questionmark’s OnDemand Service available here.

Using the Assessment Completion Time Report

The Assessment Completion Time Report is one of three new reports in Questionmark Analytics. Here are the basics:

What it does: The assessment completion time report graphically monitors how long, on average, it took each participant to complete an assessment in relation to  his or her overall score.  For example, someone who took very  little time to complete the assessment  but achieved a very high score would be flagged. Might this person have had prior knowledge of the exam (e.g., prior exposure to the answer key)? On the flip side, someone who took a very long time to take the assessment  but scored very low would be flagged, too. Might this person have been memorizing questions (i.e., taking exam content to sell to other test takers)?

Who should use it: This report provides valuable test security information for those administering medium/high stakes tests. It can also be used to help determine whether the allotted test taking time window is sufficient for most participants.

How it looks: The scatterplot shows the assessment mean for each participant (X-axis) plotted by the assessment completion time for each participant (Y-axis). Clusters of participants that stand out as having extreme combinations of assessment score and assessment completion time are evident in the scatter plot (e.g., participants with short completion times and high assessment scores would stand out in the lower right section of the graph).

Below  the scatter plot is a table that lists participants that are flagged as having suspicious combinations of assessment score and assessment completion time. This list can be useful in conducting an investigation of what happened in a given context.  Click here to learn more about Questionmark Analytics.

Streamlined certifications for a global IT services workforce

Posted by Joan Phaup

NCR automated tellers machines (ATMs) and other self-service technologies are used all over the world, and the thousands of technicians who maintain them need to keep pace with constantly changing technology as well as the general wear and tear of heavy use. NCR University, the company’s training arm, runs a global skills-based technical certification program that we first reported about in this blog a couple of years ago.

John Berry

John Berry

Since then, the program has continued to grow and prosper, so I asked for an update on the program from John Berry, who is NCR’s Technical Certification Program Manager:

Tell me a little about NCR University
NCR University (NCRU) is the destination for all employee learning at NCR. We have thousands of online technical and soft skill training courses as well as instructor-led offerings delivered in classroom settings worldwide. Our online learning catalog is constantly being updated by NCR’s Global Learning team, which I’m part of. NCR employees all over the world use NCRU every day.

How has the Technical Certification Program changed since NCR first set it up?
The processes used to create the NCR technical certification exams are the same as they were when we developed the program in 2008 in partnership with industry-recognized certification consultants. Our NCR Global Learning instructors do the bulk of the exam development, but we still ask NCR Customer Engineers (CEs for short) for input. However, we have improved the way we make the certification exams available to the CEs. In 2010, we fully integrated the Questionmark technical certification exams with NCRU and our learning management system SABA.

How has the integration changed things for you?
Before the integration, we scheduled exams manually through Questionmark. With thousands of NCR CEs worldwide, that’s a lot of exams to schedule! So the integration has really streamlined that process. Now our CEs can schedule their own tests through the NCR University. We build the exams in Questionmark and they’re available to our audience through NCRU (SABA). When someone completes an exam, we get detailed question-by-question reporting that helps make sure the exams are working properly. The detailed reports we get also help us to determine if we need to go back and further validate individual questions. So between automated registration and detailed reporting, the integration has made life easier for everyone.

How many people participate in the program around the world?
We have thousands of CEs globally, but at the moment this is a voluntary program which is supplemented with reward and recognition at each level of achievement. Technical Certification validates our CEs’ skills and abilities over and above their normal training, and we have various rewards for progressing through the increasingly difficult levels of the program. To date we’ve issued more than 4,300 certifications to more than 2,500 individuals in 49 countries, and our exams are available in up to eight languages.

How do you motivate people to take exams voluntarily?
There’s a buzz about the certification program among our CEs, because there’s prestige associated with passing the different exam levels. Those who pass the Basic Level exam receive a bronze lapel pin. We give a silver lapel pin and a customized tool bag to people who pass the Advanced Level certification, and anyone who achieves Expert Level certification gets a certificate and a beautiful watch with the NCR logo and “Expert Certified” on the watch face. We also publicize certification achievements through an online “Wall of Fame” as well as through NCR’s various communication channels.

Reward and recognition are part of this program, but the really important thing is being able to refresh and enhance our CEs’ technical knowledge. Technology changes quickly and helping them keep up-to-date with some pretty complicated hardware and software enables them be more effective at their job.

I’d like to know more about Expert Level Certification!
The ATM Expert level certification exam is currently the most difficult and sought-after exam in our program. It includes some really challenging troubleshooting questions with hypothetical ATM problems presented through a combination of videos, graphics and photos. We spent about a year gathering information from instructors and CEs all over the globe, and released the exam this September. Many CEs who had already attained the ATM Advanced Level certification were anxiously awaiting the release because they wanted to be among the first Expert Level certified engineers. And in fact, sixty of them passed the exam during the first week the exam was available!  NCR’s senior vice president for services individually recognized these CEs via personal letters, and a general announcement about their achievement was sent to the global services team. It’s a real morale booster! These CEs, who are driven to be the best they can be in their field, serve as a real inspiration to our entire field services team.

New in Questionmark Live: Question Properties!

Posted By Doug Peterson

With the latest release of Questionmark Live browser-based authoring on October 13,  you’ll see a couple of new things – the new Matching question type, the Embed Video button in the editing toolbar (to be discussed in my next blog article, coming soon!), and Question Properties. In this article we’ll take a look at Question Properties.

The first thing you’ll notice when creating a new question or editing an existing question is that the Edit Question Description button has been replaced by the Edit Question Properties button. Clicking this button will open the Question Properties window.

The Question Properties window currently contains one tab: Info. (More tabs with some very exciting features and other types of information will be coming as Questionmark Live continues to mature!)

Let’s take a look at each editable field:

  • Question Description is a short description of, or title for, the question. It defaults to the question wording unless you decide to change it.
  • Original Reference is a free-form text field intended to allow the question author to reference the source of the question. This could be a chapter of a text book, a page in a manual, or perhaps a video to which the question pertains.
  • Review Status is intended to help with tracking the question through a review and approval process. It’s a free-form text field where you could notate workflow stages such as “Draft,” “Pending Review,” “Approved,” or “Rejected.”
  • Question Status controls the delivery of the question.
    • Normal – a normal question that can be used in an assessment.
    • Retired – the question cannot be used in an assessment and is just being retained for historical purposes.
    • Incomplete – the question is still be developed and cannot be used in an assessment.
    • Experimental – the question can be included in an assessment, but it’s score will not be included in the participant’s block, topic, or assessment score. This status is useful for trying questions out and gathering statistics.
    • Beta – the question can be included in an assessment and behaves the same way as a normal question, but this value designates that it is in a beta testing state.
  • Copyright Holder is a free-form text field where you can put copyright information.

The next five fields cannot be edited. They simply provide the question type, the question’s unique ID number, the question author’s ID, the date the question was created, and the date when the question was last updated.

The SIF Association’s Assessment Life Cycle

julie-smallPosted by Julie Delazyn

While educators rely on a wide range of technologies, their primary focus and strength is teaching. Having  data systems that work together seamlessly supports them in their efforts and helps drive innovation.

The SIF (Schools Interoperability Framework) Association is working to make this happen. SIF, whose members include software vendors as well as federal, state and local educators, has been defining specifications for efficient, accurate and automatic data movement between applications.

Enhancing interoperability options between assessment management systems is an important part of this. Questionmark’s CEO, Eric Shepherd, wrote about this aspect of SIF’s work in a recent post on his own blog. He drills down into the processes involved in six key data-consuming/generating activities within the life cycle and how they relate to each other:

  • Content Development
  • Pre-Test Administration
  • Test Administration
  • Scoring
  • Reporting
  • Post-Test Administration

To read more about SIF’s assessment life cycle, click here to read the whole post. If this and other topics about assessment and learning interest you, check out Eric’s blog.

Keynote for Questionmark 2012 Users Conference: Dr. Jane Bozarth

Joan PhaupPosted by Joan Phaup

With our European counterparts having recently wound down their conference in Brussels, we are revving up for the 2012 Users Conference in New Orleans March 20 – 23!

Jane Bozarth

Dr. Jane Bozarth

Our first major announcement is that Dr. Jane Bozarth, Elearning Coordinator for the State of North Carolina, will be our keynote speaker!

Jane’s talk, Look before You Leap: What You Measure is What You Get, will emphasize the importance of planning quizzes and tests with solid objectives and clear outcomes in mind. She will demonstrate the ways in which assessments can fail to measure the right things, and she will share some tips for improving assessments by building them right into the design of a learning program.

Jane holds a M.Ed. in Training and Development/Technology in Training and a doctorate in Training and Development. She is a popular conference speaker and has written several books including From Analysis to Evaluation: Tools, Tips, and Techniques for Trainers. Jane also writes Learning Solutions Magazine’s popular Nuts and Bolts column.

Online registration is now open, with savings of $200 for those who register by December 9. Group discounts are available, too, so be sure to invite your colleagues to join you!

Experienced Questionmark users please note: The conference call for proposals is open through October 28. Case study and peer discussion proposals are rolling in, and we’d like to receive one from you, too!