Sage Pay Technical Report

Mar 14, 2011 · 3 minute read
Category: ecommerce

After the recent Sage Pay troubles, they offered a technical report which if requested they will send to you.

Not sure why they created this extra step to get to this information so I thought I would post it here for general consumption:

Sage Pay Technical Report — 3 & 4 March 2011 At 01:05 Thursday 3 March 2011 our system monitoring functionality alerted us to a potential issue with the Sage Pay Live gateway. Our Incident Management team immediately initiated our incident management processes updating the monitor page, activating engineers to further investigate the situation and engaging with senior members of the business. It was initially unclear as to the extent of the issue — Live transactions were not being processed at this time, but My Sage Pay was available. In total during this time the Live gateway was unavailable for approximately 1 hour 45 minutes. Remedial work to re-enable our transaction processing was completed at 04:25am. The nature of this work meant that transactions were not archiving to our warehouse and reporting databases. By 07:30am all transactions were able to be processed (including refunds, repeats etc), however at this point they were not appearing in My Sage Pay. We remained in this state until 11:45am. In order to try and resolve the issues with My Sage Pay, our warehouse and reporting database processes required restarting; this was performed at 11:45 but had a direct impact on our front-end web servers, resulting in a 50% reduction in transaction processing between 11:45 and 11:50. Live transaction processing on the gateway continued at a degraded level throughout the afternoon despite our efforts to resolve the issues fully. Between 13:56 – 14:10 the live gateway was unavailable. At 19:20 normal volumes of transactions through the Live gateway were resumed following the resolution of issues with the warehouse databases. The gateway has since maintained full service with no further service degradation. My Sage Pay issues continued into Thursday evening and during the day on Friday. We were unable to display real-time transactions, as the reporting databases were not being populated. We took the decision to re-index the reporting database, which began at 23:59 on Thursday in order to ensure improved performance when My Sage Pay was brought back online. Due to the issue on 3 March, this work took three times longer than expected and caused My Sage Pay to remain unavailable until 16:35 on Friday 4 March. During Saturday 5 March and Sunday 6 March further work took place to populate fraud results, correct settlement discrepancies and complete the migration of the remaining transactions. All work was complete and My Sage Pay was fully up-to-date by midday on Monday 7 March.