Restoring VM's after a host SSD gets cooked.
When I went to power my homelab server back on, after sitting idle for 8 months while we moved house, it displayed some concerning ata device warnings on boot. After a reboot it failed an automatic run of `fsck -a` and recommended performing a manual run. The root device `/dev/mapper/pve-root` was corrupted and after performing several manual filesystem checks the system could no longer boot. The 64G SanDisk SSD(SDSSDP-064G) I was using as the OS disk had shit the bed.
These are some notes from the process of reinstalling the host OS (Proxmox VE) and restoring the VM's from backup.
1. Remove the dead drive and reinstall Proxmox to a backup 128G SSD. (I've got a brand new NVME drive on the way to replace this one)
2. Mount the 2 WD Black 1TB drives I use as directory/VM disk storage to the host.
3. Recreate the config files for the VM's...
100.conf 101.conf 102.conf 104.conf
root@pve:/etc/pve/qemu-server# cat 100.conf
root@pve:/etc/pve/qemu-server# cat 101.conf
root@pve:/etc/pve/qemu-server# cat 102.conf
root@pve:/etc/pve/qemu-server# cat 104.conf
TTSH v4 Build Part 1
The TTSH (TwoThousandSixHundred) v4 is the latest incarnation of a DIY clone of the now infamous ARP 2600 synthesizer from the early 1970's. The TTSH project was originally started by The Human Comparator (thehumancomparator.net) in 2013, it has gone through many revisions and bug fixes to the main PCB since then. Originally launched it was sold as "The Kit" that came with the powder coated and screen printed front panel, main PCB and 3 VCO PCB's for $620 USD. You were still responsible for sourcing the rest of the BOM (bill of materials), the "rare parts kit" from Thonk.de or eBay and building a case for it all to fit into once done. This was a bar of entry a bit too high for me at the time and I withheld my order in hopes of a more complete kit in the future.
This latest version - v4 (rev. 8 main board) included a number of bug fixes and was made available as a complete kit by SynthCube (https://synthcube.com/cart/ttsh-v4) on June 4th 2019. I made a deal with my wife that night; I'd buy 2 of the full kits, and once completed, I'd sell one of them to "pay back" the original up front cost. On June 5th 2019 I pre-ordered 2 of the full kits, with all of the available upgrades ( all PCB's, components, metal cases, slider caps, gate booster mods, midi implants ) for a total of $3,791.23. SynthCube were also selling assembled and tested units for $3499, so I figured I would have no problem selling a modded/upgraded unit for at least as much and make most if not all of my money back.
Then started the waiting game, this was a "pre-order" after all, SynthCube gave somewhat regular updates on the progress of the panels and sourcing all of the parts. It wasn't until October 13 2019 when the kits were shipped, the first shipment only came with the panels, PCB's and parts boxes. The cases, midi implant and slider caps were still owed but I had enough to get started on the builds.
Build process to be continued...
I've setup a CloudFront distribution to serve some static assets and media files for this domain. This will make load times faster and allow me to serve content from AWS's edge locations instead of from a single container instance of this "blog".
Testing some random image embeds here:
I spent the better part of the day yesterday working on a Minecraft status checker package for the site. The Minecraft query protocol is somewhat tricky to implement for a novice like me but apparently it changes a bit when there are plugins involved on the server. I'll update this post later with some of my findings.
For now, there is a neat, albeit somewhat basic status page here: https://sqweeb.net/minecraft
Well, I tried to update this post but apparently my token went invalid while I was typing it and I lost it all when I hit "Save". I'll have to increase the JWT expiry on this server before I attempt another "large" entry update like that :(
OK, I've added some auth token refresh handling to the server and front end so I can stay logged in without issue now. I've also abandoned my own implementation of the Minecraft query protocol and decided to use a package I found on GitHub called go-mc.
This package uses the latest protocol and has a lot nicer encapsulation around packet encoding and localization that I was too lazy to bother with. It also has a much larger portion of the API implemented so I could potentially write a bot to do some player level interaction in the future. For now though, I've updated the `/minecraft` API on this server to use the new package and also keep track of players that have joined the server. Player names, uuid's, "first seen" and "last seen" time stamps are now recorded in the database. I may expand on this to include play time calculations and other fun stuff.
I've updated the `/minecraft` page to show which players are currently online, as well as changed the "Server Highlights" from cards to a carousel display. That should do it for this entry, I'll probably start a new one for further updates to the backend stuff.