DP-750 · 4.1.4 · Deploy and maintain data pipelines and workloads
Pipeline error handling
Design and implement error handling in data pipelines, notebooks, and jobs
Design for transient retries, explicit failure paths, cleanup, quality failures, and resumable processing.
Start from the failure type
- Failure is transientRetry the failed task
- A late task failed in a multi-task runRepair the run
- Cleanup must always happenAll-done finalization
- Streaming process restartsResume from checkpoint
Good recovery avoids both unnecessary reruns and endless retries.
Preserve successful work
In a multi-task job, a repair can rerun unsuccessful tasks and required dependents without re-executing every successful upstream task.
Make retry-safe work idempotent
A repair restarts an unsuccessful task from the beginning. Use overwrite, merge, or other idempotent logic when partial output could otherwise be duplicated.
Stop deterministic retry loops
Syntax errors, bad configuration, or invalid source data need correction rather than repeated execution.
Key points
- Retry: Re-run transiently failed tasks
- Run if: Route failure and cleanup paths
- Checkpoint: Resume streaming state safely
- Expectations: Decide whether bad data warns, drops, or fails
Exam trap
Blindly retrying a deterministic code or data error wastes compute and delays diagnosis.
Sources · 3 Microsoft Learn pages Show ▾
- Control the flow of tasks within Lakeflow Jobs
- Troubleshoot and repair job failures
- Structured Streaming checkpoints
Checked on October 4, 2026. Report a problem
Go deeper on Microsoft Learn
- Control the flow of tasks within Lakeflow Jobs
- Troubleshoot and repair job failures
- Structured Streaming checkpoints
Checked against Microsoft Learn on October 4, 2026. Report a problem with this card