Article: SCORM 2.0 White Paper: Stealth Assessment in Virtual Worlds

From unthinkMedia

This article talks about a new approach of using a modified SCORM (standard assessment for LMS communication) for stealth assessment in Virtual worlds. This is only a proposition, but interesting to see what researchers are thinking about regarding this issue.

I find it hard to believe that it would be possible to "standardize assessments" in games. Depending on the game, they could be so organic, especially in VIrtual Worlds. If some one deicides to take the "scenic route" to task, does that make them wrong? Game mechanics could be integrated, however not sure how well this would scale.

I am very interested in this topic so i will continue to research.

Contents

The issue

People want to have a more web 2.0 method of assessment, however the ADL (American Defense Department in charge of the standard) believes that high-stakes programs of instruction leading to certification of competence require a formal process of authentication that recreational learning does not offer.

Having this openness will enable low prior knowledge learners to choose exactly that instruction that they do not need


stealth assessment

When embedded assessments that are unnoticed by the learner, and seeped into the environment. Stealth assessment can be accomplished via

  • automated scoring
  • machine-based reasoning techniques to infer things that would be too hard for humans (e.g., estimating values of evidence-based competencies across a network of skills).

issue

  • making enough sense of the information to make inferences.
  • in traditional tests the answer to each question is seen as an independent data point. In contrast, the individual actions within a sequence of interactions in a simulation or game are often highly dependent on one another
  • methods for analyzing the sequence of behaviors to infer these abilities are not as obvious

approach

learner-players naturally produce rich sequences of actions while performing complex tasks, drawing upon the very skills we want to assess (e.g., communication skill, decision making, problem solving).

  1. evidence-centered assessment design, which systematically analyzes the assessment argument, including the claims to be made about the learner and the evidence that supports those claims (Mislevy, Steinberg, & Almond, 2003)
  2. formative assessment and feedback to support learning (Black & Wiliam, 1998a; 1998b; Shute, 2008);
  3. instructional prescriptions to deliver tailored content via an adaptive algorithm coupled with the SCORM 2.0 assessments (Shute & Towle, 2003; Shute & Zapata-Rivera, 2008a).

Evidence-centered design

Competency

What collection of knowledge and skills should be assessed?

competency describes the set of knowledge and skills on which inferences are to be based these inferences may include grading, certification, diagnosis, guidance for further instruction, etc.

Evidence

What behaviors or performances should reveal those constructs?

expresses how the learner’s interactions with, and responses to a given problem constitute evidence about competency model variables.

Questions we must answer:

  1. What behaviors or performances reveal targeted competencies?
  2. What is the connection between those behaviors and the Competency variable(s)?

Task

What tasks should elicit those behaviors that comprise the evidence?

task specifications establish what the learner will be asked to do, what kinds of responses are permitted, what types of formats are available, and other considerations, such as whether the learner will be timed, allowed to use tools

Tasks such as quests and missions, elicit evidence (directly observable) about competencies (not directly observable).

Bayesian networks

Method of figuring out probablity

  • used within student models to handle uncertainty by using probabilistic inference to update and improve belief values
  • support “what-if” scenarios by activating and observing evidence that describes a particular case or situation
  • propagates information through the network using the internal probability distributions that govern the behavior of the Bayesian net.

The Goal

  • obtain valid inferences of competencies through close examination of performance data, concept maps, and cognitive processes.
    1. global patterns emerging in the performance data and maps, as well as the cognitive processes, events, and/or conditions that trigger changes;
    2. the extent to which the changing patterns are progressing toward a target model;
    3. detailed and precise information on what and where changes are occurring within the maps
Retrieved from "http://wiki.unthinkmedia.com/index.php/Article:_SCORM_2.0_White_Paper:_Stealth_Assessment_in_Virtual_Worlds"