Automatically view and track the statistics of enemies that defeat you in Fortnite the moment you are eliminated.
Overview
This Python program combines OCR (Optical Character Recognition) and ChromeDriver
functionalities to detect when an icon is present on thes screen, then uses its position to
calculate the size and location of the
text that will be positioned relative to the icon. This text is read with an OCR, and used
to open a temporary, locally-hosted, instance of a webpage, where JSON data is stored & used
to pull data using a proprietary library I created. These values are then stored in a new
row of a .xlsx spreadsheet and the average of each column is written to the top of its
respective column. Then, the program will once again wait in the background until the player is eliminated again.
The primary intention is to have this program running in the backround
whenever you are playing, allowing the statistics to be collected over time and allowing you to see the enemy's stats briefly after being eliminated.
Functionality
Image recognition
The screen is scanned every second for crosshair icon which indicates the player has been eliminated.
The program waits until this icon is detected to begin.
Locating and processing username
Locating
The sizing and position of the enemy's username is dynamic based on its length. However, the crosshair icon's position is absolute in relation to the username. Using this information and a sufficient sample size, I was able to calculate the linear regression model of the username's dimensions and position as a percentage of the screen-width.
Processing
For more consistent results when extracting text from the screen, the screenshot taken of the enemy's username is ran through two steps of preprocessing.
1. Original image:
2. Grayscale:
3. Threshhold
Data extraction
Statistics
When the enemy's username has been extracted successfully from the processed image, a webbrowser instance of their stats page is opened. The JSON data of their stats is then obtained using the proprietary open-source library I created to do so, and used to get their playtime and KD ratio.
When the player's statistics are retrieved succesfully, they are added to a spreadsheet in the program directory and the average of the collected stats is updated in the second row.
Notable Information
Additional features
1. Function parameters are provided for the user to control if the window of the enemy's statistic page autocloses or not, and if they chose for it to autoclose, how many seconds the page remains open for.
2. A second branch is provided on Github for if the user strictly wants the webpage to open without logging any data to the spreadhseet.
OCR Training
While developing this software, I face a couple problems with extracting the username text from the screen using an OCR. The first problem was that the font used
in Fortnite is rather stylized, making it difficult for the OCR to read. Attempting to train the model using .tif line images with their respective ground-truth texts in a Linux subsystem proved to be futile.
Training models were seeing some success, but ultimately ended up skewed. There were a few specific character pairs that the OCR had trouble distinguishing (e.g. I and 1, G and 6), so I focused on training the model on
the characters that it struggled with the most. However, this led to overfitting withinin the model, developing a tendency to read those characters (this effect was notable regardless of the number of training iterations (100, 1K, 10K)).
I attempted to use Paddle OCR, however, it ended up being even less consistent. Processing the photo before passing it through Tesseract OCR (untrained) yielded surprising accurate
and consistent results
Limitations
1. While I have set up the OCR in a way that makes it reliable, it is still possible for it to misread text. Moreover, the lower-bound threshold value is 230, so values between 231-255 could be included unintentionally.
However, the threshold can be adjusted easily.
2. Usernames can include non-English characters, which arent accounted for in the chosen OCR configuration. The fail-safe in place is that the website won't be retrieve the JSON, and will return nothing.
3. I haven't yet tested if the software works when playing Fortnite on resolution that isn't the monitors native resolution. This may cause issues when detected the crosshair icon asset.