DeepLeague: leveraging computer vision and deep learning on the League of Legends mini map + giving away a dataset of over 100,000 labeled images to further esports analytics research
GitHub Repo + Dataset: https://github.com/farzaa/DeepLeague
Note: All of this is free + open-source. I explain all the gritty technical details in Part 2 of this post which you can find here. Feel free to contact me at anytime if you have questions.
Note 2: If you’re an LCS Team hire me! I’ll help you beat our Korean Overlords. Contact me on Twitter :).
For a moment, imagine you are the coach of a soccer team. Now imagine that someone gave you a computer program that takes a video of a soccer game and output the coordinates of your team’s players at every moment of the game. If we visualized that data it would look something like the GIF below where you can see the players on the blue team have little boxes drawn around which the program produced. The program “watched” the game for us!
Do you see how this information would be extremely useful for a soccer coach? By having this data, you could do an endless amount of analysis on the movement of your players. For example, if you wanted to improve your team’s defense you could analyze the movement of your players whenever the opposing team scored a goal and create new strategies + fill weaknesses.
So, can this same concept be applied to video games?
HELL YEA IT CAN BOIZ LEMME SHOW U.
What is DeepLeague?
DeepLeague is the first algorithm and dataset (over 100,000 images) that combines computer vision, deep learning, and League of Legends to move LoL analytics to the next level by giving developers easy access to the data encoded in the pixels of the game. It specializes in predicting bounding boxes + labels for champions given just an image of the League of Legends mini map. So, all you give DeepLeague is a VOD (which is basically just a bunch of images) and it will output something like this:
It uses deep convolutional neural nets and is trained on a portion of a very large dataset of about 100,000 labeled mini map images (see GIF below) that I programmatically created and am giving away for free. Silicon Valley A.I companies pay tons of money to get labeled data. I don’t have money but sometimes throwing money at a problem isn’t necessarily the best solution. Sometimes you just need to be a little clever!
Other then the dataset itself, I’m giving away my actual methodology for how I created the dataset which you can read all about in Part 2. I want A.I research in esports to grow and that starts with developers endorsing it. I hope that by giving away the dataset/methodology, people become a bit more interested :)
TLDR: DeepLeague “looks” at the mini map and tells you where every champ is at every moment during the game without any other input.
So, given just a video, you can aggregate the champion and the champion’s position at every single second during the game. There are many use cases for this, especially in the LoL esport scene where this data would be very useful, because all you need to give DeepLeague is a VOD of a player’s game. This means it works for scrims, solo-q games, and other personal games where all that’s required is a simple VOD (even ARAM!).
Before DeepLeague, there was no way to get this kind of in-game data since the Riot API only provides post game analytics.
Here are a couple of things you’d be able to do given the output of DeepLeague.
- analyze how the jungler paths, where he starts his route, when/where he ganks, when he backs, which lane he exerts the most pressure on, when/where mid roams
- analyze when teams set up dives, when they decide to do dragon, how they rotate around the map as a team, how they set up baron
- analyze when laners overextend, when laners die, when they are getting pressured by other lanes, when they are losing lane, when they get solo killed, when they leave laning phase
- AND OF COURSE. You can run DeepLeague on VODs of Faker playing to aggregate data about how he plays and truly learn from the best :)
Lets talk about it.
Why did I make it ?
I’ve been heavily into esports for around four years now and one of the most interesting things about it for me is all the work that happens behind the scenes of a pro-team. Pro-players have an insane amount of support from their team in the form of coaches, analysts, chefs, fitness trainers, psychologists, and more so that the player has nothing to worry about but the game itself. But, the industry is very young and the methods being used right now may not be the best. In order to illustrate this, I’d like to go over some of the key responsibilities of a full-time League of Legends Analyst:
- watch hours and hours of competitive gameplay from various regions, like Europe and Korea, to get a solid grasp of the evolving meta before it evolved so they could stay ahead of the other teams.
- take notes on champions that seemed to be overpowered on a specific patch so that they could take advantage of these “power picks” during their own games.
- take notes on countless other things like ward placements, jungle routes, when teams took dragon, when teams grouped, how teams set up dives, where players died, where they were overextending, etc.
This goes for every single hardworking coach/analyst in the scene. The only way to make sure your team has the best chance of winning is by first gathering all this information, understanding it, and then figuring out how to feed the info to the players who will be in the driver seat during the actual game. I’m sure you can see that not having a coach or analyst for a professional League of Legends team is borderline heresy!
But, the job description for an analyst sounds quite painful. Worst of all, because the industry is so young, the best tool that exists for an analyst at the moment is Microsoft Excel.
Now that you know a little more about some stuff that happens behind the scenes, let me talk about why more useful tools do not exist.
One of the most important things to understand is that Riot Games doesn’t have any sort of API that gives you information about what happens second by second during the game. All they give is the post game statistics like kills, deaths, etc. This SUCKS for people, like pro League of Legends teams, who would really benefit from such detailed live game info. There is a non-permanent workaround to this that I’ll explain later, but it only works for live LCS games. Lets take a look at a possible situation.
Lets say that TSM scrims another team, like Cloud 9. The players would be at their respective gaming houses playing on their personal computers. After the game ends there is no way to figure out what happened during the actual game without actually watching the VOD again. Where did he die? Where did he ward? Where did he start his jungle path? All this info has to be manually aggregated. Some people might say “why can’t you aggregate data directly with the client?”, well 1) it’s against the terms of service and 2) its encrypted and especially difficult to crack.
The Minimap
I knew immediately that I wanted to use the minimap for this little project. It provided the most information about the game in the simplest manner in terms of the complexity of the image. This is a very good relationship for your dataset to have, especially when utilizing deep learning, because it means your algorithms will probably have an easier time learning.
The minimap also provides an alarming amount of information related to the the state of the game as well. If a human looked at the mini map just once every minute, they’d have a pretty good understanding about who’s winning, how many structures were destroyed, placement of wards, etc.
The next step was figuring how the heck I’d actually aggregate this data via computer vision.
Can I Train a Deep Neural Net to Predict Bounding Boxes + Classify Champions on the Minimap?
I’ve used deep neural nets in the past for tasks related to self-driving cars and its obvious that they are extremely powerful. I know some readers of this article may not understand how neural networks work or what they’re capable of, but let me show off a little example to help you out.
Below is a little GIF from a neural network I created + trained for a car to predict steering angles (you can check out my work with self-driving cars here). I came up with my own neural net architecture and trained it on about 8 hours of driving data provided by the awesome guys at Comma AI (shout-outs to my boy eder, my #1 inspiration for all things A.I). Every single image within the dataset is associated with a steering angle.
With just 8 hours of data, the neural net magically learned some of the basic rules of the road without ever being told so. The black dot is what the actual steering angle is. So this would be the angle that the driver who was originally controlling the vehicle had his steering wheel at. The red dot is what my neural network predicts the steering angle to be based on what it sees in the image.
So, you can barely see the black dot because its usually being covered by the red dot. This is good! It means our neural network is doing a very good job at predicting how to steer the vehicle based purely on the pixels in the image. Thats fucking DOPE.
I want to show off one more neural net because I think the concepts relate heavily to the ideas behind DeepLeague. Neural nets can also be used for object detection. I’ve seen tons of different models that specialize in object detection such as SSD, R-CNNs, Faster R-CNN, and YOLO9000. For one of my projects, I needed to perform object detection in real-time on a dash-cam video. This meant that as a person drove around, this neural net would be saying stuff like “hey thats a person” or “hey thats a car”. I decided to go with YOLO9000 because it was the fastest detector around, but not necessarily the most accurate!
Below is a GIF of a pre-trained YOLO model I used to detect objects in real time from a dash-cam video.
So now the question is: can deep neural nets that are used for object detection be trained to recognize objects from video games?
Hm. It makes some sense though, right? If we can train a neural net to understand what objects within complex scenes look like in the real world, we should be able to train it on video games objects! Using this basis, I went off to create the dataset.
