Using the Dragon Age Toolset to Conduct Experiments

Hey all, sorry for the lag in posts. I have been slacking off my duties, but my excuse is that James and I have an exciting new experiment in the works. As part of the experiment we are using the Dragon Age Toolset, and through working with it extensively over the last few weeks I am very impressed by the power that it provides to an experimental designer interested in games. Most researchers in academia don't have direct access to a legion of designers and engineers, so it is great to find a tool that can be used to create experimental environments fairly quickly.

So what is it that makes the Dragon Age Toolset so great for experiments? Well, there are a few features that stand out, but this post will focus on experimental control.

Over the last 4-5 years one of my biggest concerns with games research was the inability to perform true controlled experiments. Those who are familiar with the research know that there are a wide variety of experimental studies that have examined video games. Of these only a handful have actually been able to manipulate the environment in a controlled fashion. Typically, what I have found is that there are three general levels of control that exist in experimental research.

No Control

The first of these I will categorize as no control. These are studies that attempt to evaluate the influence of an independent variable on a dependent variable, yet use games that have differences beyond the independent variables being manipulated. An example of this would be measuring the recall of aggressive words as a dependent variable, having aggressive content as the independent manipulation, and then having one group of players play Tetris and the other group play Counter Strike. The problem here is that Tetris and Counter Strike are different in many ways beyond the experimental manipulation.
Now, this doesn’t mean that this research was bad research. It simply is a reflection of the tools that were available at the time. This kind of research has certainly lead us to interesting questions, but it cannot establish a causal relationship between a set of variables, or validate a theoretical process.

Control Under Designed Constraints

The second type of research that I want to identify has experimental control, but is limited in the questions that can be asked because it must draw from the pre-designed features of games. An example of this type of research might examine the influence of in game communication on subsequent behavioral measures (such as prisoners’ dilemma strategies and payouts). In this type of experiment the researcher uses a single game that has been designed with different customizable features to conduct the experiment. The number of teammates in FIFA 11 or the presence or lack of voice chat among teammates in halo would be examples of manipulations in this kind of research.

Under these conditions experimental manipulations are possible. An experimenter can control for all variables (except the independent) across conditions, but unfortunately is limited in the questions that can be asked. This also places constraints on the theories that can be studied, because the appropriate customizable feature may not exist, or may be slightly off what is required. Some adjustable features map perfectly onto theory; however, more often it is the case that they don’t quite match up with the theory and so researchers still can’t properly validate the process of interest.

Controlled Manipulation Through Design and Engineering

Currently there are only a handful of studies that are able to achieve controlled manipulation of a game environment where the manipulations are specifically designed with a given theory in mind. Perhaps the earliest study in this paradigm was performed by Sheese and Graziano, in which they compared the influence of playing Doom II (with and without monsters) on cooperative behavior (they used an alternate of the prisoners dilemma). Now, the argument can be made that removing the monsters from doom is not a good operationalization of violent content. Perhaps, varying the number of monsters across multiple levels (high, medium, low, none) would have been a more accurate portrayal of the influence of violence on cooperative behavior. Regardless, what Sheese and Graziano did was the right way to go about doing experimental manipulations in video games. They used game design to construct a protocol and operationalize their concepts and then engineered an environment to test their theory.

This is exactly the kind of power that the Dragon Age Toolset provides and it does so in a fairly easy to use and very powerful package. With the Dragon Age Toolset an experimenter who has a minimal amount of coding skill and is willing to spend some time learning the toolset can manipulate things like number of monsters, the price of goods, the conversations with other NPCs, or the facial features or voice of the players avatar – these are just the tip of the Ice Berg. My point here is fairly simple. Up until now experimenters interested in the influence of game play on attitudes, beliefs, and behavior have used either different games or asked questions that were convenient for a particular game design. The Dragon Age Toolset, and its older sibling Neverwinter Nights both provide researchers with the ability to design games that target the questions that they want to ask.

It is my hope that researchers will see the value of these tools and that game designers will continue to release games that have built in capabilities for editing. Dragon Age is powerful, but you still can’t turn it into a customizable first-person shooter or driving game.

Using the Dragon Age Toolset to Conduct Experiments by Travis Ross, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

No Responses to “Using the Dragon Age Toolset to Conduct Experiments”


  1. Motivate. Play. - GDC Tuesday: Playdom on Behavioral Economics in Games - [...] doesn’t seem to exist – or exists but only under certain conditions. Interestingly, the Dragon Age experiment that I …

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>