In my last post, I outlined the few things that I was going to focus on in order to stop feeling so overwhelmed. Well, I’ve made some progress.
The two items I focused on first were the deployment script and streaming data. The deployment script may sound like a trivial thing that should’ve existed long before now, but cut me some slack. The streaming data is a bit more complicated, and will always be a continuing project.
Deployment Script
As you learned in my last post, if you even read it, I’ve just been manually deploying random anime all of these years. It’s a statically generated site, so deploying my UI is just FTPing the files to the server. I’d then hop over to phpMyAdmin and export/import tables. It worked, so don’t judge.
The reason for wanting a deployment script at this stage is purely for convenience. The FTP process takes a long ass time, so I’d rather just have the script handle it. The script itself is fairly simple, but I got to use some node command line packages that I had never used before (chalk, inquirer), so that was fun.
The script, as you may have figured out from the previous sentence, is in JavaScript, ran locally using Node. I have it separated into API, UI and data. I can deploy all three, or choose.
The API was pretty straight-forward. It’s written in PHP. No fancy frameworks or build process. It’s literally just FTP the files up. It’s quick and easy.
The data was fairly simple as well. There are only like 10 or so tables. The terrifying part would be accidentally deleting data. So it creates the MySQL dumps and saves them for backup.
UI was a little weirder. It’s Nuxt, so there’s a build step. The generate command for Nuxt has always been squirrely for me, so I had to run that command in a loop in case it randomly fails like it does sometimes. Then the files are copied over to a temporary directory.
After that, I created a loop to prioritize things. Images, which I can opt out of, goes first since it affects nothing but takes a bit. Any files that aren’t in the anime or like directories goes next. These will be all the dynamically created pages and generated payload files. This goes fairly quick and gets the update to the majority of the site.
The anime and like directories go next - because there is a fuck ton of them. Every anime gets rendered within anime, and those with like data are rendered in like. So.. it’s a lot. Takes a while. So it goes last.
Streaming Data
This is like my fourth or fifth iteration of how I scrape my data from the streaming giants. I wish I could just hit an API for this, but any I have found either cost money or don’t update as often as I’d like. Especially for newer shows.
This iteration is a browser extension that I load into my browser locally. I have scripts for each of the sites I want to scrape. They open up in each tab in my browser and go on their merry way. Some are quick (Hulu, Tubi) and some take FOREVER (Netflix, Crunchyroll).
I like displaying the audio language for my streaming links, I think it is super useful to know. Because of that, for like Netflix or Crunchyroll, the scripts have to index all of the shows and then go to each, individual page in order to determine the audio selection. Pain in the ass.
Shoutout to HiDive’s absolute div soup that makes it a pain in the ass. I even have to scroll freakin’ carousels to get things to load. Crunchyroll isn’t much better - their browse dynamically loads/unloads shows as you scroll. Fun, right?
Anyway. When they are done, they download a JSON of all the data they’ve collected. Sweet. Next step is matching that data to anime that I have on the site. Historically, this has been a pain point. The titles have slight variations that make everything a pain in the ass.
I think my process as it is now is pretty good. First, exact match. Easy. Second is a fuzzy match where high enough ratio is auto-matched, then slightly below that is reviewed by me with an easy Y/N answer. I’ve conceptualized a third step that prompts a local LLM, but I don’t think it’s needed at the moment.
After that - it does a validation check to remove any duplicates that it may have matched. This is, again, semi-manual as it’ll prompt me with options for me to choose the correct title, or none.
Once the matching is done, there is a simple script that updates the table responsible for the matching. Done!
Next Steps
So, what’s next now?
According to my list from before, I think I want to take a stab at a script that monitors the anime trailers. After streaming data, the trailers are the next thing that change frequently - and by change, I mean they break.
That’ll probably be my next move. If not, it’ll be a script to more easily mass-update scores/episode counts.
Thanks for reading!