Novelcrafter inaccessible due to infrastructure upgrade

This incident has been resolved.

Here's what happened:

At 12:21 UTC, we made an upgrade of the infrastructure to Novelcrafter. The changes we implemented were:

1. Switching from x86 to an ARM-based architecture
2. Switching from 'buffered' responses to 'streamed' responses.

Internally, updates are done via automation that add a new version with the changes, and then roll over the instances/servers one by one. Usually this causes no issues, except taking a bit of time.

However, the switch to a streamed response had to be done asynchronously, and not in time with the rest. This caused the responses from Novelcrafter confusing a down-stream service about the format - returning incorrect content.

We tried to rollback the upgrade, but just as it finished, the down-stream service swapped over to the correct format - causing the same issue, just in reverse.

The issue was resolved when all server instances were back up-to-date with today's update.

At 12:31 UTC the services were operational again.

The issue has been identified and a fix is being implemented.