DP-750 · 3.4.4 · Prepare and process data
Pipeline expectations
Manage data quality with pipeline expectations in Lakeflow Spark Declarative Pipelines
Use Lakeflow expectations to declare data rules and choose whether violations are monitored, dropped, or fail the update.
See the expectation outcomes
Lakeflow expectations combine a Boolean quality rule with a defined response when the rule is violated.
Choose warn, drop, or fail
Warn
- Invalid row continues
- Yes
- Pipeline continues
- Yes
- Best fit
- Observable issue
Drop
- Invalid row continues
- No
- Pipeline continues
- Yes
- Best fit
- Safe rejection
Fail
- Invalid row continues
- No
- Pipeline continues
- No
- Best fit
- Critical invariant
Warn records quality metrics while allowing data through. Drop excludes invalid rows. Fail stops the affected update.
Keep recoverable data recoverable
Use a quarantine pattern when rejected records must be inspected, corrected, or replayed later.
Monitor the evidence
Expectation metrics and pipeline event logs show violations over time and help troubleshoot failed quality rules.
Key points
- Expect: Records violations while allowing processing
- Expect or drop: Removes violating rows
- Expect or fail: Stops the affected update
- Metrics: Pipeline event data exposes quality results
Exam trap
Use fail only for critical invariants; overusing it can make pipelines unnecessarily brittle.
Sources · 2 Microsoft Learn pages Show ▾
Checked on October 4, 2026. Report a problem
Go deeper on Microsoft Learn
Checked against Microsoft Learn on October 4, 2026. Report a problem with this card