[Post Mortem] Insufficient support on dropped columns in multi-table rivers

Summary

On Jun 25th, 2020, an issue was raised about insufficient support on dropped columns in multi-table rivers. The issue had been reproduced, fixed, and deployed on the 28th of Jun 2020.

Root Cause

The is was a result of the updating metadata mechanism in multi-table rivers, which did not synch dropped columns in the source table in Rivery internal metadata service. The issue, in this case, made data from tables with dropped columns in the middle of the table columns list, and their data type after the dropped fit to the target data type, to be shifted and was not updated correctly.

Resolution

  • Correspondent from Rivery checked out the issue and determined it as high-priority.
  • In order to continue working, metadata from these specific cases were dropped and re-inserted to the internal metadata service.
  • The fix changed the way the metadata service updates the columns definition in the backend.
  • The fix was deployed on June 28th, 2020.

Improvements

First, we apologize for the inconvenience caused by this incident and make sure it won’t happen again.

The improvement made by the fix, which now ensures the metadata is updated correctly in the system backends.

Thank you,

Rivery Team

Note:
The information contained in this report is confidential and is intended solely to promote safety and reduce customer risk.