(Time in PST until I convert)

12:47

• We are back up… again! All testing infrastructure is disabled, and investigation is still ongoing

• You might be asking why we would need to “refresh” (delete) the database for our tests, and the answer is this screenshot. We make a new fake test user for every test, so that we can make sure that each use case is taken care of (is the user staff? gold account?). And we don’t want the results to be polluted by what other tests do, so we generate tens of thousands of users for these tests. Having those running alongside our regular database would make the site come to a massive crawl.

  

12:20PM

• Something mysteriously fired off the DB job again 😡 I have a suspicion that it’s coming from an old Github job that was supposed to run the tests and bring the site down if they didn’t pass. I’m investigating that path now while restoring the database to before they fired.

• I’m also pushing up a change to remove all tests from our repository until this is resolved 😢

• Very funny to still be able to do this in 2023 😬

  

10:45AM

• Had to give [new.subeta.net](<http://new.subeta.net>) (and the Subeta API) an additional little kick to get restarted. We use a system of environment variables that gets passed to each of our individual sites, including the information for the database connection. The API did not restart after getting the new data, so was just running off the old DB. Fixed now!

9:35AM

• Database has been fully restored, and I’m doing final checks now to make sure everything is where it should be 👀. I’ll then be doing a full audit of the mechanism that fires off tests before enabling them again. Thank you for your patience this morning!

9:15AM

• Waiting for the database to restore, could take another 15-30 minutes. We’ve got minute by minute backups turned on this database so that we can restore to any point in time in the last 7 days. We’re pretty protected by incidents like this, unlike 17 years ago when I deleted the first version of Subeta by accident 😉