Game Analytics - An Interview with Dr. Dmitri Williams

While I was at GDC Austin back in October (I have been side tracked by a dissertation) I was fortunate enough to meet and interview Dr. Dmitri Williams who is a cofounder and CEO of the analytics company Ninja Metrics and a Professor at USC. Below I have transcribed the interview where I ask Dr. Williams about the relationship of his company to his social science work and his thoughts on analytics.

Hope you enjoy.


Travis Ross: So I was curious. How did you get started with the company? What led you to your venture?

Dr. Dmitri Williams: So my colleague Jaideep Srivastava, the cofounder of my company actually. We were sitting in the Washington DC airport after doing a grant session trying to get money for our research projects, and he said, “You know, what if we turn this into a business?” And I said, “What if we turned what to business?” He said, “Well we are doing all this predictive modeling and all this analytical work. I’m sure that there must be some kind of commercial application.” I said, “That’s kind of crazy, who would, you know, be interested in that kind of stuff?” And I sat there and thought about it. And I thought, “Okay, I see that it would work really well. I just don’t want to do it. I want someone else to do it.” He said, “Well there is no reason we could not do it and get it up and running and then bring in people who were better at it to do it. If it’s an opportunity in a space that no one has done anything then we should think about this.” I said, “Well that’s pretty interesting. How about we talk about it after I get tenure?” And that’s essentially how it happened.

As soon as I had the ability to do it with a little less risk to my academic career I went ahead and did that very thing. We took it more seriously over time. We started thinking about who would work there. What  products we would sell. Where we would get the venture capital. The management plan. The business plan. Would we have a sales force? All of these things. There was a recognition at the outset that it is a totally different skill set than what I have trained to do over the last 10 years in academia. But, I actually do have some background in business and marketing way back during my undergrad life, so it was not completely foreign. All the same, I think it was important to recognize at the outset there was a lot of stuff that I didn’t know and that I would have to be aware of my ignorance to some extent. There would be a learning curve. A big learning curve.

TR: Are you using the data and findings from the company in your own research? Or are those two things completely separate from one another?

DW: Oh. No, those two things cross-pollinate quite a bit. So, a lot of the ideas that we have for our company operations come from best practices in academia. That part is really obvious. Different people were doing a lot of predictive analytics work on one grant project for various intelligence agencies, which are using virtual world data and using big machine learning algorithms to try to predict who will do what next. That kind of runs along the same lines as models of who’s spent or who’s going to buy, at what stage, and when/why. So, for example, when you run a machine learning algorithm you don’t always get an answer that makes the same kind of sense that a regression line would, or even the kind social science output you are used to, because it’s a different format. I had to kind of get used to that process and understand how to interpret decision tree output as opposed to a regression. So, I learned one and that is the one that we ended up using on the commercial side. At the same time with our funding agency I have to educate them on how to understand the results. So if they didn’t understand what the things were, I would say, “Well here is the information gain and so on.” All these things I had educated myself on with my computer science colleagues I had to turn around and educate our sponsor on. I realized, well they’re kind of like the end-user and they will be like game companies. They don’t necessarily want to know how I am doing what I am doing. They just want to know a simple answer and if it is actually working. I can turn that into an output that makes sense commercially for them rather than what would be good for a research paper. That kind of translation that is not trivial thing to have to do. To think about what they are thinking/wanting at kind of a meta-level and understand what format that would be the right level of trade-off between power and understandability. I could get something very powerful and predictive, but they literally would not be able to understand it. There are models that no human being would be able to understand the output, because it is simply too complex to be interpretable. These are really powerful models, but is that actionable? Well, yes some circumstances, and no and others. So, I had to talk to the people in these businesses a lot and go to GDC and figure out  what they want. I just came from a session on modeling where a guy was standing in front of a crowd of people essentially trying to educate the audience on the things that I know about. Saying, “This is what predictions are.”, “This is what variables are.” You look around the audience and ten percent of them are like yeah I get it. Eighty percent of them are like what the hell are you talking about. And ten percent are going to the next session. We have to be aware of what we can offer, so my function wound up being translator quite a bit, because I was bringing together two disciplines with computer science and social science and now I’m selling it to an audience that does not understand it either. So I have to be like a bridging figure this is a really challenging function.

TR: Interesting. Cognitive science has become interested in the black box of the mind. How do you see what you are doing with analytics contributing to social science theory and that aspect of things?

Well the real challenge to theory is that the best model is completely theoretical. So if we are in social science and you are looking at R2 or whatever it is that determines if you have a good model. Let’s say that R2 gets to be like .4. That is really good and most papers are published with R2 around .05 and that is pidilly. The models are not that good, but because they are theoretically sound and are statistically significant they get published. So I go over to my computer science colleagues and they say, “Oh I am sorry this model is not very good. It is only predicting with like 75% accuracy.” I am like, “Jesus Christ.” Okay, so this model is like 10 times better than our published models. I say, “Let me see the output.” And ask, “What does this mean?” They say, “Well we don’t know.” Nobody could actually understand it. So I say, “Okay, so there is no connection to theory?” and they say, “What’s theory?” Computer scientists to worry about theory they don’t have multiple co-linearity problems. They throw as many variables in as they like and the result is, that there is a model that is so complex that at the end the only use is that you could run it in reverse and deliver a very accurate  pen point calculation for any one person. For example, for that guy in the crowd there he is 85% chance that he will get divorced tomorrow, and like crazy stuff like that. But, they have no idea why. So there is a tension. Do you want a model that has theory and is understandable and interpretable? Okay that is going to be less powerful. Or do you want a hard-core model that is dead on accurate, but you have no idea why. From a social science perspective that is a very intense trade-off and I am sure it is going to be a big issue. People will probably write papers on it, but really we’ve only been alert to this in the last year and a half or so and our colleagues don’t even know what I’m talking about. On the operations side for a company they have the same issues. I could say, “Well do you want a result that says this person is X% likely to churn? Or do you want one that has less power, but you know why you need to intervene?” It is the same issue, the academic and the corporate actually have a lot in common that both want to understand it but both have to respect the power. To be honest we are just learning. It is something that the guy last session was able to tap dance around or even not bring up because it’s not a flaw necessarily, it’s an unknown. The thing is the science is so cutting-edge there are no best practices yet because the practices are as much about how management or the people in the academy used these findings as they are about the methods or the findings in the first place. You can have the best model in the whole world but if nobody can use it then it’s not really the best model.

TR: The industry seems to be focused very heavily on the term engagement. Are you all dealing with trying to operationalize this concept and present it to companies?

Well I don’t think anyone can do it all that well with metrics. As a social scientist I like to use survey data and that sort of thing. Anything that is behavioral is a proxy for engagement and it is weak.  You can say this guy is interesting because he is playing with us in this pattern and it predicts that he drops off tomorrow, yet you don’t know why. And the reason is he wasn’t engaged, but you don’t know why. Your model tells you hey he’s go drop off because of his behavior pattern, but you never know what is going on with him psychology because that data is never in the model. There is nowhere to capture that in behavior not even a rough proxy. This is a flaw until we can find awesome proxies for these behavioral states or attitudes, or we can put in survey variables. That is always going to be a weak spot in the model.

At this point the line to see/hear Neil Stephenson started moving and Professor Williams and myself concluded the interview to jump into the line.

Game Analytics - An Interview with Dr. Dmitri Williams by Travis Ross, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

2 Responses to “Game Analytics - An Interview with Dr. Dmitri Williams”

  1. Thanks for posting this. I've been trying to find a concise sort of "excuse", if you will, for the use of analytics in the game industry and elsewhere. Williams did a great job at clearing up how it works and why its used.

    In facebook games, you give access of your profile to a company, and the company gets to learn real life information from you and can use that data for their models of your behavior. But what about games like SWTOR, WoW, EVE, and other games for which personal information is more sparse? I suspect these companies use your in-game behavior to predict whether you're going to keep paying for the monthly subscription, but they definitely don't get a "why?" out of that, which would allow them to address attrition. I think that's where the study of human motivations as they relate to gaming really becomes useful to industry.

    • Travis L Ross says:

      Isaac, when we went to GDC Austin Bioware gave a really remarkable presentation regarding the use of analytics in SWTOR. The are using them heavily to help tune the world. The have an entire tool set that over lays a really wide variety of behavioral measures that might be useful to designers. Essentially they are creating visualizations of how players interact with the world. They have a fairly large team of analytics folks working on just figuring out what kind of behavioral measures can be represented visually to aid the tuning of the world and game designers in finding places where there are bottle necks. It was a really cool presentation and I hope that I can find out more about it this year in San Francisco.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>