Advisory on Gevme Email Sending System Disruption
When: June 20th - 25th 2024
Issue Overview:
The disruption originated from the Apache Kafka messaging service, which is integral to transferring emails from Gevme to the mailing system. Following a security patch applied to the AWS managed service, the Kafka messaging service began resending previously processed emails to the mailing system.
Impact:
This unexpected behaviour resulted in the reprocessing of some past emails, which in turn caused a delay in the processing of newly created emails. Consequently, a queue of new emails built up while the system was occupied with reprocessing older emails.
Resolution:
To address this issue, we have implemented enhanced checks on campaign processing to prevent duplicate processing and have recreated the Kafka messaging service cluster. These measures have successfully resolved the disruption and restored normal email processing operations.