I may have over-corrected with Donkey Kong Bananza, but I did find some time to do some additional coding. When I updated Royfuss Games to 3.0, I ran all of the scripts, except for the upload scripts. The upload scripts determine which game and achievement pages to upload by comparing the active database data to a copy of a cached version of my database. They upload only the games/achievements pages that change. After uploading, it copies the active database as the cached copy. Since every page changed, I just manually uploaded them.
This week was the first time running all scripts, including the upload scripts. All scripts worked great, but I had some test data in the cached copy of the database. It uploaded an additional page that it didn’t need to. It was not a big deal, until I noticed I forgot to program logic to copy the active database to the cached database afterwards. Oops. It was an easy fix, and I was confident everything would work next time.
I wanted to update one more thing on Royfuss Games. I wanted all of the “blog mentions” for the games to point to Royfuss Archives, instead of the original posts. This meant changing links for over 400 games. This data is stored in a CSV file, so it is easily manipulated with Python. My Royfuss Archives scripts need to do a similar link translation, and spits out a CSV file with all of the original post links and the Archives links. So, I used Python to go to work using these two files. I ran into some issues where I had to tweak the data. For example, I had a bad link for one of the “blog mentions.” Other than that, I was done with all of that in under an hour.
I decided to run my Royfuss Games scripts again with the new link data. This was a good test for my upload scripts. It correctly identified all 400+ game pages that needed to be uploaded, uploaded them, and created the new cached database. I should have been done with Royfuss Games for awhile.
Or so I thought. I have a separate set of scripts to get my achievement data and the achievement icons. I talk about the process a bit here. Originally, I was not going to update them, but I guess I still had the programming itch. I ended up redoing all the scripts.
In that post, I mentioned RetroAchievements has a public API, but it was offline. It is available now. It has been available for long time, but I never got around to updating my script. Instead, I did a lot of manual work. I navigated to the website, logged on, selected the game, viewed the page’s source, and copied and pasted the HTML to an offline HTML file. I did it this way, because I was not able to figure out how to login to a site with Python to do some webscraping. So, I just webscraped using the offline file I copied. It worked great, but it did take a little bit extra time.
I’m a lot more confident now with Python. I could get the authenticated page to scrape the page, but the API is the better option. My new scripts will save me a bunch of time. The script displayed the last 10 games I played, and I just select the game, and it outputs my achievement data into a CSV file, and downloads the icons. From there, I just need to import the data into my data base, just like I do with my other scripts. Also, I copy over the icons to the correct directory, so they can be uploaded.
I decided to take another crack at my Xbox script. Xbox does not have a public API, so I had to use a third party. When I last tried, I attempted to use OpenXBL. OpenXBL lets you make 150 requests/hour for free, which is plenty for me. Like an idiot, I did not look at my old script before attempting to write the new one. If I would have, I might not have tried again.
The new script started well enough. OpenXBL has separate API calls for getting Xbox 360 data and newer Xbox’s. It required a little more code, but It was easy enough to implement. Soon enough, I was able to get most of the achievement data I needed for my Xbox games. I saved the achievement icon image for last. That’s when I discovered why I dropped OpenXBL the last time.
Xbox 360 games do not have icon images. They just don’t. For some reason, Microsoft uses 1080p images for their icons for the rest of their consoles. I need the images 64×64, so 1920×1080 is problematic. First, the images take forever to download. I ran it for Psychonauts 2, which has over 50 achievements. The script took 15 minutes to run. Second, I then have to manually crop the images to a square, and then I can resize them down to 64×64. For a game like Psychonauts 2, the badge I want is actually centered on the image so this is easy. However, for games like Gears of War: Ultimate Edition and Halo: Master Chief Collection, the images are pretty much screenshots; they do not look good as icons. I script creates my CSV file I can import, and downloads the images (at least for later consoles), but I rarely use it.
Instead, I re-wrote my Exophase scraping script. I needed this script for EA achievements and others, so using it for Xbox is not a big deal. Exophase uses a combination of APIs and webscraping to get their data. They do not want people to scrape their data, which I find funny. Because of this, I have do something funky. First, I have to navigate to my Exophase page. Then I have to click on the game. With the game’s achievement page open, I can see all my achievement data, but I can’t scrape it directly. I can’t even see the data if I view the page’s source. I found a way around it, though. I have to highlight the entire achievement table, and then view the selection source. In this source, I can see my data! I paste it to an offline HTML file, and then perform my scraping on this offline page. It works similar to how used to do RetroAchievements. It is able to create the CSV file, and download the icons. Good enough.
Since I was deep in it, I trudged onward to my PlayStation trophies script. As mentioned in “Scraping Achievements,” I had to use PowerShell to get my authentication token to use the API. Now that I have a better understanding of how to use Python with APIs, I wanted to give it another shot. I thought I coded something solid, but it didn’t work. I turned to AI, and it’s logic seemed sound. When I implemented its code, it, too, failed. The parameters, headers, and other things needed just did not work with Python’s Requests for some reason.
I decided to stick with my PowerShell script to get the JSON files needed to get my trophies. I did tweak some things to make the script more efficient. Now, it checks to see what game and trophy data I already have downloaded, and only downloads new. This will reduce the number of requests I use, which is important, because the API blocks requests after a certain number in an amount of time. I wrote another Python that allows me to delete my offline game and trophy JSON files if I need to update a game. I have another Python script that allows me to search the saved offline files to generate the CSV file, and download the trophy icons The process works well, and I have it well documented.
Steam was the last of the scripts I updated. I did not have to change much of the logic, but one thing bothered me about my existing scripts. For hidden achievements, I could not get the achievement description. I have been manually copying and pasting the data from my achievement list. This was annoying. I did another sweep, and could not see a way to get it via the API. So, I did some searching online. I ran into several old threads asking the same question, and an answer popped up. On January 11th of this year, someone resurrected these threads with one comment: https://api.steampowered.com/IPlayerService/GetGameAchievements/v1/?key=***********&appid=220&language=english&format=xml. For those who are not API savvy, the key needs to equal your Steam API key, the appid needs to equal the game’s appid, the language should be whatever language you need, and the format can be either xml or json. The “localized_desc” field is the achievement description. Steam doesn’t have this documented, so I am not sure if this will change, or what they will do with it. I’m happy for now.
I think I am done with Python scripts for now. We’ll see if I keep that promise; I do want to update some things for my remastered “Games of” posts. For now, I’m going back to Donkey Kong Bananza.