Data Quality

Approach

Maintaining correctness of data and features is a top level design goal for Fennel.

While Fennel has best-in-class diagnostic and monitoring levers too, unlike many other systems out there, Fennel's approach leans heavily on preventive measures that prevent failures from happening in the first places.

Here are some of the key ideas that help prevent/diagnose data quality issues:

TypeMethodDetails
PreventiveStrong TypingLink
PreventiveImmutability & VersioningLink
PreventiveUnit TestingLink
PreventiveCompile time lineage validationLink
PreventiveStructured metadata & ownershipLink
DiagnosticData ExpectationsLink
DiagnosticFeature Drift DetectionLink

Each of these methods is already powerful on their own. And their preventive/diagnostic power further amplifies when applied together.

Edit this Page on Github