Issues exporting product feeds

Incident Report for Norce [Jetshop]

Postmortem

During the day on December 5th we got alerted that our product feeds in Hyperdrive were not updating.

Based on the messages in our logs it was clear that the database had been corrupted due to a safety server restart.
To fix the issue we had to run a full import of products.

However, because of the load, our memory storage instance started to dump a lot of data on the disk and filled it up. This caused the service getting down time in the evening at about 5.30 PM EST.

To mitigate the issue we added more memory to the server and increased the limit for the memory storage instance.

The service was restored at 8.45 AM EST as of today December 6th.

Posted Dec 06, 2023 - 12:04 CET

Resolved

This incident has been resolved.
Posted Dec 05, 2023 - 22:30 CET

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Dec 05, 2023 - 22:05 CET

Identified

The service is partially down, we have identified the issue and we are working on resolving it.
Posted Dec 05, 2023 - 21:28 CET