the creative cost of analytics

I work in the analytics industry as an engineer who facilitates the deployments of (primarily) predictive systems. I essentially never talk about my job on this blog because it’s a) something personal that I don’t want everyone to know about, and b) I want to talk about TTRPGs and video games on this blog, not what I do on a day-to-day basis. In this instance however, there’s a good crossover between the two as more and more industries leverage analytics in decision-making. While most analytics implementations are going to be so invisible to the average person on the street, more and more you’re going to see the impact of analytics on creative industries. You are already seeing the impact of it due to the rise of platforms like Netflix, where the ability to create new content has been combined with the ability to precisely analyse the consumption patterns of the audience. Netflix knows what shows and films you watch, it knows what genres are the zeitgeist, and they are able to create new content to fit into those gaps. While the pandemic and general destruction of the cinema-going experience has certainly accelerated the move to streaming platforms, I think it’s more to do with the position of data as “digital gold” that has caused them to duplicate like rabbits. Increasingly, companies want to know what they’re doing will be a success before they do it, and analytics has given them some of the tools needed to achieve that. Where this intersects with the gaming industry is fairly obvious: they’re also doing it, just not as effectively.

Before I go about this, let me do some defining of terms. While machine learning is a gigantic deal at the moment, and you might think that almost all innovation in analytics over the last ten years has been in “artificial intelligence”, you’re probably wrong. For almost any company, the outputs of machine learning systems will be utterly dwarfed by the machinery that supports more general analytics. These are your dashboards, your data pipelines, your mundane triggers (where a certain customer event triggers a certain response), your non-ML traditional statistical analytics and capabilities that we normally throw under the umbrella titles of “business/data analyst”. Yes, Netflix does have a formidable collection of machine learning systems that power things such as their reccommender engine (what Netflix decides to show you when you open the webpage), but they’ll almost certainly have an equal or greater number of people asking questions like “what was the most profitable content we put out in the last year?” or “what was the greatest return on investment that we had across all content types?”. These are not questions that necessarily require ML systems to answer, they can be answered using very basic stats and data querying. All of this activity has been enabled by new capabilities and enhancements in the data collection and processing technology stack. The industry has increased the cadence of this tremendously, going from weekly, to daily, to real time (as it happens) analytics. While the production time of content for Netflix will make a lot of these mechanisms slower (they can’t just produce an entire film to fill a gap that opened up yesterday), it is still going to accelerate.

This is not just the case for streaming platforms. Across essentially all creative industries, we know more about what does and does not work. If you go and look at a title like Doom (1993), despite the game essentially creating an entire genre and being possibly the most well known game ever made, the wake that follows it is on a significant delay. In 1996 you had Duke Nukum: 3D, and Blood in 1997, forming what a lot of people consider the holy trinity of the early FPS genre. Into the 2000s, the FPS genre would continue to see rapid innovation, growing to be the premiere game genre, possibly tied with the third-person action game purely through Grand Theft Auto. You start seeing monumental titles like Halo, Quake, Unreal Tournament, etc. The point I’m making here is that it actually took a fair amount of time for FPSs to be realised as an absolute moneyspinner; sure, people knew that Doom and Wolfenstein were good, but you only started seeing saturation in the genre later on. Contrast this with the Battle Royale genre, where the timeline of major releases looks something like this:

  • March 2017 PlayerUnknown’s Battlegrounds enters Early Access

  • September 2017 Epic releases Fortnite: Battle Royale

  • February 2019 EA releases Apex Legends

  • March 2020 Activision releases Call of Duty: Warzone

If you had asked me in mid-2019 if I thought there was any more room in the Battle Royale genre, I would have said no. For a considerable amount of time, the “holy trinity” was PUBG, Fortnite and Apex Legends, and sure while there was some absolute bottom-feeder titles such as the Paladins battle royale mode, the foundations had essentially been set. I feel that, had CoD Warzone been released by another company, it probably would have been forgotten to time without the immense weight that the Call of Duty title affords it. While there have been a few efforts here and there in more recent times (such as Spellbreak), the general sentiment seems to be that the genre has reached saturation. Big name developers don’t seem to be exploring the space further beyond maintaining those existing titles, an attitude almost certainly spurred on by the catastrophic failure of Ubisoft’s Hyper Scape. So in a three year period, we saw the release of the title that lit the fire (PUBG), followed by what is probably the last big name entry in Warzone. Am I saying that we’ll never see the battle royale genre come back? Absolutely not, nobody would be stupid enough to make that claim; maybe there’ll be some advancements in tech that enable new and exciting iterations, sparking a “second generation” much in the same way that we had generations within the FPS genre. At the moment though, it seems like it has reached peak saturation, and new additions are coming from the indie/small dev scene.

You probably know where I’m going with this, but why did it take so long for the FPS genre to start to reach saturation, even after (what was at the time, the eponymous title) the biggest entry into it was released, while the battle royale genre was saturated within three years? Was it the development times of those older FPSs? Nope, in fact, the development times of those older games are (sometimes) hilariously short, with Blood’s development being publicly announced in 1996 (with a 1997 release, development following the success of Duke Nukem 3D). My hypothesis is that the reason we’re seeing such tighter cycles of genre generations is because we simply know when something is a success far faster. You could argue that the battle royale genre started with the mod in ArmA III, which had success in 2012 followed by the release of the official DayZ mode. While there’s pretty stark differences between DayZ and the modern suite of battle royale titles, we can still think of this in terms of generations. The ArmA III mod roiled away with considerable popularity for a considerable length of time, yet we saw no huge mainstream uptake by developers/publishers. I’d wager this is because a) they didn’t know if it was popular simply because it was a free mod and b) at the time, those numbers would not have been as obvious and available (or possibly even used in earnest by those companies). Yet the moment we have a paid-for title hit the deck, at a time where the numbers (player counts) are public knowledge, we see the entire industry spring into action. Rinse and repeat for the card game genre, the hero shooter genre, the autobattler genre.

Previously, we’ve seen companies that release sequels for titles in quick succession. You might use Blood being released so soon after Duke Nukem 3D as being evidence that we were already developing things in that way. I’m not trying to argue that previously, companies had no idea what was successful or what wasn’t. Of course 3DRealms knew that Duke Nukem 3D did well, they got paid the money from those sales. What has changed is, because these numbers are now more publicly available, and the speed at which we learn those numbers has massively increased, the performance of a game has essentially become public knowledge. We knew that PUBG was a success the moment it released, because you could look at the steam player counts and see that it was a success. All you had previously were things like the NPD sales figures, which had a (relatively) slow cadence, and were (comparatively) informed estimates. We went from a period where only the company that released the game truly knew the figures behind it (though these would be announced if it was a publicly owned company), to a member of the public being able to do back-of-the-envelope (number of players) x (RRP of game) calculations. This is all enabled by the machinery of data, the consolidation of game platforms, and the movement away from retail platforms like brick and mortar stores. Even if you buy a game from an actual physical store now, whoever owns your console of choice will be told that you played it. Even if you bought it for PC, Windows (via the Xbox App) will almost certainly be using that information.

The title of the blog post is the ‘creative cost’, so what is that cost? I harp on and on in this blog about how companies hate risk. This is why analytics is like company crack, because it’s an extremely good tool for mitigating it. Companies are able to make increasingly informed decisions about what they’re putting out onto the market, with tools that are more concretely able to say “you should expect X return on Y product release”. Obviously in the past, companies didn’t just vomit random products onto the market because they didn’t know what worked, but there was far more human interaction and far more latency in understanding what did and didn’t. This yields space for things like experimentation. The original release of Star Wars sparked an entire generation of sci-fi copycats trying to cash in on its ludicrous popularity, but where that process spanned years, now it spans months. Is this a bad thing? Well, creatively, yes. A big limitation of these analytics-driven development approaches is that they’re typically descriptive. They can tell you that PUBG was a success, and they might even be able to tell you why, but if PUBG didn’t exist, what would they say? Imagine a bizarre world in which we had all of the data and analytics machinery that we have today, but Doom hadn’t been made. Would any of these systems suggested that we develop Doom? I’m sure someone could call me out here and say theoretically yes, but practically the answer is almost certainly not. As people familiar with analytics can tell you, this is the leap between descriptive to prescriptive analytics, which is vast. We have excellent tools and historical success in the descriptive field, but success in the prescriptive is far harder. If you’ve ever wondered to yourself why your Youtube recommendations seem so terrible, this is why. Incredibly easy to tell what you’ve watched, and possibly even why you watched them, but far more difficult to suggest what you should watch next.

So what does a world in which descriptive analytics systems dominate business decision making lead to? Well, it leads to a world in which Netflix outputs a steady stream of identical military action films. It’s a world in which the mobile game market, where development time and cost is incredibly small, is a homogenous nightmare-scape. Will companies take risks? Absolutely, the chance that they’ll happen onto a black swan that’ll make ridiculous sums of money (Squid Game, comes to mind) is always there, but otherwise the main bulk of programming will be essentially algorithmically curated. Youtube is a vision of the future, where the huge “development” capacity and audience has accelerated this process a thousand fold. In some horrid inversion, Youtube’s fate seems to be less algorithms describing current consumption patterns, and more creators attempting to identify the consumption patterns of the algorithms, but the outcome is the same. Identical thumbnails, title structures, content. If you create something refreshing and new, it might blow up, sure; more likely you’re just creating something that hasn’t been quantified and featurised yet, and that’s a problem. Our ability to model success, or future impact, is very frequently a question of how much history there is to draw upon. By their definition, novel creations do not have history, and so we are tremendously disincentivised from creating them. This is a different problem from the issue that Youtube’s algorithms face (where a massive feedback loop occurs from creators making content that appeals to the algorithm, and the algorithm pushing views to videos that satisfy its internal calculus), but it is nonetheless a huge constraint of creative ability.

I can tell you this as someone who implements analytics solutions: we do not want novelty. When I’m working on a task and the data changes significantly, or new classes of thing start appearing, that’s a problem. We don’t like surprises, we don’t like unique things, we want certainty and categorisation — these things are the antithesis of creative thinking, and why this stuff is so dangerous.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s