Articles:TRUE Instrumentation: Tracking Realtime User Experience in Games
From unthinkMedia
A system of automating a collection of gameplay feedback that is streamed for the entire duration of the the game. This makes it possible to collect large amounts of data efficiently, and is paired with other data streams such as video and in-game surveys. This allows researchers to understand what players are actually doing in the game, and if there are any catalyst of unintended behaviors.
Contents |
Playtesting
This is a great way to test games, however you loose granularity Typically, playtest result in identifying big problems, but the nuances may go unnoticed.
TRUE how it came about
It is expensive to watch people at the level of detail needed to come out with valuable results, and identifying underling causes of any issues that arise.
The idea of TRUE started as nn external application where players would press buttons in order to log various game states. This was a manual process which wasn't extremely valid since many times players would forget to self record them selves killing a boss, finishing a level, etc.
Having a hard time understanding how a company like Microsoft would actually think that self recoding would be a good idea, while simultaneously playing a game.
The external processes where quickly integrated into the code through hooks, so there was no longer a need to manually report.
One it was clear that this would be a great method of measurement there were a series of requirements that where collected.
- Collection of attitudinal feedback via a in-game surveys
- Collection of contextual data as well as the main data or interest.
- Utilizing captured video to better understand the data
Features
In-game survey
Include brief surveys, in-game.
- Event based surveys
- On Demand Surveys: User could call up a survey at any time.
- Time-Based Surverys: Good for mapping out enjoyment over time. One thing to be careful with, is not to interrupt important game play, so they are held from dispatching until there is a break in game play.
Contextual Data
These survey help with giving any relevant contextual data. In order for a survey to be actionable we need to understand the context of the answer. "It's not fun" is not enough if you don't know the context surrounding the comment.
Two pieces of content that always get attached to each piece of data
- Build number
- Test Name: Some times there are multiple test in the same build.
- Participant ID
- Timestamp
- Difficulty setting
- Chapter (name/mission)
- Position (x/y)
Lessons Learned
- Plan for instrumentation early, make sure there is time for iteration
- Start with your research questions
- Keep the number of variables you are tracking to a minimum
- Build sample reports BEFORE you set your hooks.
- Represent data visually
- Have designer specify their designs intent so it could be compared with the actual behavior
- Test to make sure you data is being recoded
- integrate in the source tree.
- Instrumentation does not replace other forms of getting feedback
Video
Screen recoding are synced with the automated contextual data, making it easier to click to certain areas of a video w/o countless hours of analysis.
