-
Notifications
You must be signed in to change notification settings - Fork 1.8k
feat: Add DELETE/UPDATE hooks to TableProvider trait #19142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
7505171 to
695716c
Compare
Rebased on latest main. Fixed clippy warnings in tests and updated delete.slt/update.slt expected output to reflect new DmlResultExec physical plans. All tests pass locally. Could you please re-trigger the CI? Thanks! |
Add infrastructure for row-level DML operations (DELETE/UPDATE) to the TableProvider trait, enabling storage engines to implement SQL-based mutations. Changes: - Add `DmlCapabilities` struct to declare DELETE/UPDATE support - Add `TableProvider::dml_capabilities()` method (defaults to NONE) - Add `TableProvider::delete_from()` method for DELETE operations - Add `TableProvider::update()` method for UPDATE operations - Wire physical planner to route DML operations to TableProvider - Add helper functions for extracting filters and assignments This provides the API surface for downstream projects (iceberg-rust, delta-rs) to implement DML without custom query planners. A reference MemTable implementation follows in a subsequent PR.
Clippy warns about assertions on constants being optimized away. Changed to assert_eq! for explicit value comparison.
- Change assert_eq!(x, true/false) to assert!(x)/assert!(!x) per clippy - Update delete.slt and update.slt expected error messages to match new physical planner behavior for unsupported DML operations
22c7273 to
b038d04
Compare
Rebased on main. Previous CI failures were due to clippy bool_assert_comparison warnings and mismatched expected error messages in test files. Requesting a CI trigger to confirm I've successfully stopped embarrassing myself. 🙏 |
Add infrastructure for row-level DML operations (DELETE/UPDATE) to the TableProvider trait, enabling storage engines to implement SQL-based mutations.
Changes:
DmlCapabilitiesstruct to declare DELETE/UPDATE supportTableProvider::dml_capabilities()method (defaults to NONE)TableProvider::delete_from()method for DELETE operationsTableProvider::update()method for UPDATE operationsThis provides the API surface for downstream projects (iceberg-rust, delta-rs) to implement DML without custom query planners. A reference MemTable implementation follows in a subsequent PR.
Which issue does this PR close?
delete_fromandupdateinTableProvider#16959Rationale for this change
DataFusion parses DELETE/UPDATE but returns NotImplemented("Unsupported logical plan: Dml(Delete)") at physical planning. Downstream projects (iceberg-rust, delta-rs) must implement custom planners to work around this.
What changes are included in this PR?
Adds TableProvider hooks for row-level DML:
Physical planner routes WriteOp::Delete and WriteOp::Update to these methods. Return type matches insert_into(): Arc producing {count: UInt64}.
Are these changes tested?
Yes. Unit tests for DmlCapabilities:
cargo test -p datafusion-expr -- dml_capabilitiesA follow-up PR provides MemTable implementation with comprehensive sqllogictest coverage.
Are there any user-facing changes?
New trait methods on TableProvider:
Fully backward compatible. Defaults return NONE or NotImplemented.