Skip to content

Conversation

@ethan-tyler
Copy link

@ethan-tyler ethan-tyler commented Dec 7, 2025

Add infrastructure for row-level DML operations (DELETE/UPDATE) to the TableProvider trait, enabling storage engines to implement SQL-based mutations.

Changes:

  • Add DmlCapabilities struct to declare DELETE/UPDATE support
  • Add TableProvider::dml_capabilities() method (defaults to NONE)
  • Add TableProvider::delete_from() method for DELETE operations
  • Add TableProvider::update() method for UPDATE operations
  • Wire physical planner to route DML operations to TableProvider
  • Add helper functions for extracting filters and assignments

This provides the API surface for downstream projects (iceberg-rust, delta-rs) to implement DML without custom query planners. A reference MemTable implementation follows in a subsequent PR.

Which issue does this PR close?

Rationale for this change

DataFusion parses DELETE/UPDATE but returns NotImplemented("Unsupported logical plan: Dml(Delete)") at physical planning. Downstream projects (iceberg-rust, delta-rs) must implement custom planners to work around this.

What changes are included in this PR?

Adds TableProvider hooks for row-level DML:

  • DmlCapabilities struct with NONE, ALL, DELETE_ONLY, UPDATE_ONLY constants
  • dml_capabilities() returns supported operations (default: NONE)
  • delete_from(state, filters) deletes matching rows
  • update(state, assignments, filters) updates matching rows

Physical planner routes WriteOp::Delete and WriteOp::Update to these methods. Return type matches insert_into(): Arc producing {count: UInt64}.

Are these changes tested?

Yes. Unit tests for DmlCapabilities:

cargo test -p datafusion-expr -- dml_capabilities

A follow-up PR provides MemTable implementation with comprehensive sqllogictest coverage.

Are there any user-facing changes?

New trait methods on TableProvider:

fn dml_capabilities(&self) -> DmlCapabilities;
async fn delete_from(&self, state: &dyn Session, filters: Vec<Expr>) -> Result<Arc<dyn ExecutionPlan>>;
async fn update(&self, state: &dyn Session, assignments: Vec<(String, Expr)>, filters: Vec<Expr>) -> Result<Arc<dyn ExecutionPlan>>;

Fully backward compatible. Defaults return NONE or NotImplemented.

@github-actions github-actions bot added logical-expr Logical plan and expressions core Core DataFusion crate catalog Related to the catalog crate labels Dec 7, 2025
@ethan-tyler
Copy link
Author

Hi! This is my first contribution. Could a committer please help trigger CI? This addresses the request for DELETE/UPDATE support.

cc @alamb @adriangb

@ethan-tyler ethan-tyler force-pushed the feat/tableprovider-dml-hooks branch from 7505171 to 695716c Compare December 7, 2025 18:29
@ethan-tyler
Copy link
Author

Hi! This is my first contribution. Could a committer please help trigger CI? This addresses the request for DELETE/UPDATE support.

cc @alamb @adriangb

Rebased on latest main. Fixed clippy warnings in tests and updated delete.slt/update.slt expected output to reflect new DmlResultExec physical plans. All tests pass locally. Could you please re-trigger the CI? Thanks!

@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Dec 8, 2025
Add infrastructure for row-level DML operations (DELETE/UPDATE) to the
TableProvider trait, enabling storage engines to implement SQL-based
mutations.

Changes:
- Add `DmlCapabilities` struct to declare DELETE/UPDATE support
- Add `TableProvider::dml_capabilities()` method (defaults to NONE)
- Add `TableProvider::delete_from()` method for DELETE operations
- Add `TableProvider::update()` method for UPDATE operations
- Wire physical planner to route DML operations to TableProvider
- Add helper functions for extracting filters and assignments

This provides the API surface for downstream projects (iceberg-rust,
delta-rs) to implement DML without custom query planners. A reference
MemTable implementation follows in a subsequent PR.
Clippy warns about assertions on constants being optimized away.
Changed to assert_eq! for explicit value comparison.
- Change assert_eq!(x, true/false) to assert!(x)/assert!(!x) per clippy
- Update delete.slt and update.slt expected error messages to match
  new physical planner behavior for unsupported DML operations
@ethan-tyler ethan-tyler force-pushed the feat/tableprovider-dml-hooks branch from 22c7273 to b038d04 Compare December 8, 2025 00:45
@ethan-tyler
Copy link
Author

Hi! This is my first contribution. Could a committer please help trigger CI? This addresses the request for DELETE/UPDATE support.
cc @alamb @adriangb

Rebased on latest main. Fixed clippy warnings in tests and updated delete.slt/update.slt expected output to reflect new DmlResultExec physical plans. All tests pass locally. Could you please re-trigger the CI? Thanks!

Rebased on main. Previous CI failures were due to clippy bool_assert_comparison warnings and mismatched expected error messages in test files. Requesting a CI trigger to confirm I've successfully stopped embarrassing myself. 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate core Core DataFusion crate logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support delete_from and update in TableProvider

1 participant