The rollout of the fix to the data pipeline has been completed. We will be monitoring closely for any further issues with data integrity and data latency over the next day before closing out this incident.
Posted Sep 10, 2020 - 23:49 MDT
Update
GoSpotCheck engineering will be rolling out our fix to the production pipeline starting at 6pm mountain time tonight, continuing until approximately midnight. During this deployment and rollout window, expect to see occasional delays in delivery of up to two hours in duration (manifesting as stale data in reports).
Posted Sep 10, 2020 - 16:38 MDT
Update
The GoSpotCheck Engineering team is currently running comprehensive data integrity checks against the fix that has been developed for the data pipeline, in preparation for production deployment.
The complete rollout of the fix to production is expected before the end of this week. We will provide updates as we continue to make progress towards that end.
Posted Sep 10, 2020 - 12:51 MDT
Identified
GoSpotCheck is currently working to resolve an issue with inconsistencies in the data pipeline that is responsible for populating the data warehouse used by the Insights reporting platform. These inconsistencies manifest as temporarily missing or a delayed arrival of data in reports.
While working to get a permanent fix in place, we are currently mitigating this issue by performing full resynchronization of key tables into the warehouse on the cadence of four times a day. This mitigation restores any data that was previously missing or inconsistent.
GoSpotCheck understands that maintaining the integrity of the data we capture on behalf of our customers is of critical importance. We appreciate your patience and understanding as our engineering team works diligently to implement a permanent solution. We expect to have full resolution in the coming weeks, and will continue to post relevant information and status updates on our progress here.