Game developers have allowed gamers to share their achievements for decades. In the arcades, players could enter their name or initials next to their high score. Now, platform holders have integrated achievements into their systems. I don’t consider myself an achievement hunter, but I do like this feature of modern games. I just wish it was easier to share them.
As mentioned in my previous post, I put in a lot of hours implementing achievement tracking into my Royfuss Games database and website. I wanted to make sure the achievement title, description, the image icon, and my unlock information was represented. When I set out to do it, I had three reasons.
One, I wanted to be able to preserve my achievement data. Braid is a clear example. There is no unlock date and time data on my Steam page for the game. There are a few other examples, including EDF 2017 on Xbox. I want to be able to preserve the data, even if the platform holders take it down, or if an unlock glitched.
Two, I wanted to showcase achievements that are not publicly shared. EA, Ubisoft, Rockstar, etc. do not have a way to share achievement data. I wanted a place to showcase those achievements, along with the others.
Third, I want to make sure my achievements can be publicly available. This is related to one. Right now, Steam is the only platform where you can directly link to achievement data for games. Exophase, is able to utilize API’s to share most of the achievement data from other platforms, but I wonder how long that will last. If the platforms begin to close off their API’s, sharing achievements data might become even harder.
It turns out getting achievement data is already difficult. I wanted to use scripting to get the achievement data into a CSV format, so I could easily import it into my database. I also wanted it to download the icon files. There were a lot of hoops I had to jump through just get my own data. I suspect privacy plays a key role. I think privacy is important, but I do wish the platform holders would make it easier to opt-in for sharing achievement data.
Steam was the easiest. They have a fully documented web API available to anyone who applies. I was able to get setup with an API key. I had to toggle the option on my account to allow API access. I already had that enabled, so it was not a big deal to me. From there, I was able to use Python scripts to scrape my achievement data and download icons. I am glad it was so easy, because Steam has the most achievement data for me.
PlayStation has the second most achievement data for me, and proved a bit more difficult. PlayStation has an API, but it is not publicly documented. Luckily, other people have documented it. The messiest part was getting an authentication token. I was not able to make it work with Python, but I used the Powershell 7 option, and it worked great! I ended up using Powershell to authenticate, and download all of the data into JSON files. After I had the JSON files, I used a separate Python script to get the data into CSV format, and download the icons. It is not too bad, but I have to remember to re-run the Powershell script to refresh the JSON data.
I thought Xbox achievements would be easier. Microsoft was the first major platform to offer achievements. They used to be publicly available. I think they also used to have a documented API. However, that is no longer the case. I found companies selling API’s, but no freebies. I think these companies have found a way to scrape the Xbox website for data in combination with using your login credentials. I think that’s how Exophase and TrueAchievements does it. I do remember having to login to my Xbox account, and giving them permission to access it.
I do not have a ton of Xbox achievement data, so I found a hacky way to do it using Exophase. You can scrape the achievement title, description, and even download the icons from for Xbox achievements on the Exophase page for the game. However, grabbing the unlock data requires a manual hack. You can view the unlock data on the page tied to your Exophase ID, but you cannot scrape the data by script. For some reason, it doesn’t show up in the source code for the page. So, I go to the page, and copy it into a text file. Then I have a Python script that will scrape the page for the achievement data, and then it will read the text file and find unlock information. It is extremely hacked together, but it works great! I do the exact some thing for EA achievements.
RetroAchievements is interesting, because they have a public API. However, it is currently broken. What a time to be scraping achievements from their service. I found another hacky way. As far as I can tell, you have to be logged in to view your unlock data for achievements. I login, browse to my profile, and click on the game I want. Then, I save the source HTML to a local HTML file. I tell a Python script to process the file. I do not have a ton of RetroAchievement data, so this was not too bad. I will revisit it once they finish overhauling their website, and their API works again.
For Ubisoft and Rockstar, I do not have a lot of achievement data. I used a script to get the title, description, and download icons from Exophase (for a different platform, like PlayStation), but I manually enter the unlock information. The script still saves me a lot of time.
I am working on achievement pages for Blizzard games. I only have two: Starcraft 2 and Diablo 3. However, both of them have hundreds of in-game achievements. I attempted to use the Battlet.net API. I created a client ID, generated a client secret, and got OAuth working. I was able to get my account ID number, so I know it was working. However, I could not get any other API requests to work. I then found out that you could only get Starcraft 2 achievement data, but not Diablo 3‘s, so I gave up after half a day. Instead I am doing it manually. I have taken screenshots of all of the in-game achievement pages, and I am going through them one-by-one, adding the data to CSV files, and cropping out the icon. I did run a script to scrape a fandom wiki for Starcraft achievements, but the data hasn’t been reliable. At the very least, it was initialized my CSV files. It will probably take me a few months, since I do not want to spend all of my free time doing it.
That is my adventure into achievement scraping. It was a hard process, but I also had a lot of fun figuring things out. Adding or modifying achievement data is a relatively quick process, so I think the work was worth it.