Past Incidents

This incident has been resolved.

Here's what happened:

At 12:21 UTC, we made an upgrade of the infrastructure to Novelcrafter. The changes we implemented were:

1. Switching from x86 to an ARM-based architecture
2. Switching from 'buffered' responses to 'streamed' responses.

Internally, updates are done via automation that add a new version with the changes, and then roll over the instances/servers one by one. Usually this causes no issues, except taking a bit of time.

However, the switch to a streamed response had to be done asynchronously, and not in time with the rest. This caused the responses from Novelcrafter confusing a down-stream service about the format - returning incorrect content.

We tried to rollback the upgrade, but just as it finished, the down-stream service swapped over to the correct format - causing the same issue, just in reverse.

The issue was resolved when all server instances were back up-to-date with today's update.

At 12:31 UTC the services were operational again.

Help Center / Documentation is operational again.

There haven't been any further issues or customer support tickets about this error. As such, we mark this incident as resolved.

API is operational again.

OpenRouter is operational again.