feat: Support multiple external table locations by kumarUjjawal · Pull Request #22695 · apache/datafusion

kumarUjjawal · 2026-06-01T11:18:33Z

Which issue does this PR close?

Part of #16303.

Rationale for this change

CREATE EXTERNAL TABLE can reference only one location today. This adds support for listing multiple explicit locations and reading them as one table.

What changes are included in this PR?

Adds LOCATION ('a.parquet', 'b.parquet') syntax.
Keeps LOCATION 'a,b.parquet' as a single path, so literal commas still work.
Carries multiple locations through the logical plan and proto.
Updates listing table creation to scan all listed locations.
Requires all listed locations to use the same object store and matching fields.
Keeps stream tables limited to exactly one location.
Updates docs and upgrade notes.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes. CREATE EXTERNAL TABLE now accepts a parenthesized list of locations.

There is also a public API change: CreateExternalTable.location is replaced by CreateExternalTable.locations.

github-actions · 2026-06-01T11:42:45Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v53.1.0 (current)
       Built [  93.866s] (current)
     Parsing datafusion v53.1.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion v53.1.0 (baseline)
       Built [  93.158s] (baseline)
     Parsing datafusion v53.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.588s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 189.204s] datafusion
    Building datafusion-catalog v53.1.0 (current)
       Built [  36.015s] (current)
     Parsing datafusion-catalog v53.1.0 (current)
      Parsed [   0.025s] (current)
    Building datafusion-catalog v53.1.0 (baseline)
       Built [  36.053s] (baseline)
     Parsing datafusion-catalog v53.1.0 (baseline)
      Parsed [   0.026s] (baseline)
    Checking datafusion-catalog v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.139s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [  73.194s] datafusion-catalog
    Building datafusion-cli v53.1.0 (current)
       Built [ 159.455s] (current)
     Parsing datafusion-cli v53.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion-cli v53.1.0 (baseline)
       Built [ 158.927s] (baseline)
     Parsing datafusion-cli v53.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-cli v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.122s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 321.128s] datafusion-cli
    Building datafusion-expr v53.1.0 (current)
       Built [  25.640s] (current)
     Parsing datafusion-expr v53.1.0 (current)
      Parsed [   0.072s] (current)
    Building datafusion-expr v53.1.0 (baseline)
       Built [  25.447s] (baseline)
     Parsing datafusion-expr v53.1.0 (baseline)
      Parsed [   0.072s] (baseline)
    Checking datafusion-expr v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   1.080s] 222 checks: 220 pass, 2 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CreateExternalTable.locations in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:219
  field CreateExternalTable.locations in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:219

--- failure struct_pub_field_missing: pub struct's pub field removed or renamed ---

Description:
A publicly-visible struct has at least one public field that is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/struct_pub_field_missing.ron

Failed in:
  field location of struct CreateExternalTable, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/7d27e3cd4b05927eb5e5c15355a876bf2ea9f0bf/datafusion/expr/src/logical_plan/ddl.rs:215
  field location of struct CreateExternalTable, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/7d27e3cd4b05927eb5e5c15355a876bf2ea9f0bf/datafusion/expr/src/logical_plan/ddl.rs:215

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  53.219s] datafusion-expr
    Building datafusion-ffi v53.1.0 (current)
       Built [  57.022s] (current)
     Parsing datafusion-ffi v53.1.0 (current)
      Parsed [   0.055s] (current)
    Building datafusion-ffi v53.1.0 (baseline)
       Built [  57.496s] (baseline)
     Parsing datafusion-ffi v53.1.0 (baseline)
      Parsed [   0.059s] (baseline)
    Checking datafusion-ffi v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.234s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 116.085s] datafusion-ffi
    Building datafusion-proto v53.1.0 (current)
       Built [  55.443s] (current)
     Parsing datafusion-proto v53.1.0 (current)
      Parsed [   0.017s] (current)
    Building datafusion-proto v53.1.0 (baseline)
       Built [  55.644s] (baseline)
     Parsing datafusion-proto v53.1.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-proto v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.255s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 112.514s] datafusion-proto
    Building datafusion-proto-models v53.1.0 (current)
       Built [  23.583s] (current)
     Parsing datafusion-proto-models v53.1.0 (current)
      Parsed [   0.119s] (current)
    Building datafusion-proto-models v53.1.0 (baseline)
       Built [  23.393s] (baseline)
     Parsing datafusion-proto-models v53.1.0 (baseline)
      Parsed [   0.120s] (baseline)
    Checking datafusion-proto-models v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   1.560s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CreateExternalTableNode.locations in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:250
  field CreateExternalTableNode.locations in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:250

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  50.217s] datafusion-proto-models
    Building datafusion-sql v53.1.0 (current)
       Built [  39.805s] (current)
     Parsing datafusion-sql v53.1.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-sql v53.1.0 (baseline)
       Built [  39.723s] (baseline)
     Parsing datafusion-sql v53.1.0 (baseline)
      Parsed [   0.031s] (baseline)
    Checking datafusion-sql v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.229s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CreateExternalTable.locations in /home/runner/work/datafusion/datafusion/datafusion/sql/src/parser.rs:255

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  80.813s] datafusion-sql
    Building datafusion-sqllogictest v53.1.0 (current)
       Built [ 161.032s] (current)
     Parsing datafusion-sqllogictest v53.1.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v53.1.0 (baseline)
       Built [ 161.075s] (baseline)
     Parsing datafusion-sqllogictest v53.1.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.089s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 324.952s] datafusion-sqllogictest

martin-g · 2026-06-01T13:25:09Z

 }

+fn schemas_have_same_fields(left: &SchemaRef, right: &SchemaRef) -> bool {
+    left.fields() == right.fields()


https://github.com/apache/datafusion/pull/22695/changes#diff-8ce463e0d56c67a237fce2942f0759984d56764b13b62156472132ab84c78192R226-R227 says Schema metadata may differ between files ... but here the equality check compares the fields' metadata too - https://docs.rs/arrow-schema/58.3.0/src/arrow_schema/field.rs.html#113
Is this intentional ?

martin-g · 2026-06-01T13:36:31Z

    #[prost(message, optional, tag = "9")]
    pub name: ::core::option::Option<TableReference>,
-    #[prost(string, tag = "2")]
-    pub location: ::prost::alloc::string::String,


What is the lifetime of a serialized CreateExternalTableNode ?
If there is a use case where a CreateExternalTableNode serialized with DF v.X and deserialized with v. X+1 then it would be better to keep location around until v. X+2 (assuming users don't upgrade from X to X+2).
I.e. deserialize should do:

if locations is not empty then use it

otherwise if location is not an empty string then insert it to locations

Serialize should: 1) write non-empty locations and location=""

martin-g · 2026-06-01T13:47:00Z

                            "CreateExternalTable",
                        )?,
-                        create_extern_table.location.clone(),
+                        create_extern_table


If backward compatibility is needed, i.e. if a CreateExternalTableNode is serialized with v. X and it has to be read with v. X+1 then you need to keep location around for at least one major version.
If such compatibility is not needed then you can just use "" here. It is overwritten by .with_locations(...) below

martin-g · 2026-06-01T14:04:48Z

+                        {
+                            return plan_err!(
+                                "All locations of a CREATE EXTERNAL TABLE must have the \
+                                 same schema, but the provided locations have differing schemas"


This error message does not help the user to find the problematic schema/location.

martin-g · 2026-06-01T14:07:06Z

    ) -> Result<Arc<dyn TableProvider>> {
        Ok(Arc::new(TestTableProvider {
-            url: cmd.location.to_string(),
+            url: cmd.locations.first().cloned().unwrap_or_default(),


This could silently use "" if no locations are provided. Better return an error.

martin-g · 2026-06-01T14:09:10Z

+            }
+            write!(f, ")")
+        } else {
+            write!(f, "LOCATION {}", self.location)


I see that this is how it worked before but shouldn't the locations be quoted ?

kumarUjjawal · 2026-06-02T05:11:54Z

Thank you @martin-g for great feedback. I have addressed the concerns.

feat: Support multiple external table locations

a088be8

github-actions Bot added the auto detected api change Auto detected API change label Jun 1, 2026

martin-g reviewed Jun 1, 2026

View reviewed changes

ignore schema metadata

24e71e1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support multiple external table locations#22695

feat: Support multiple external table locations#22695
kumarUjjawal wants to merge 2 commits into
apache:mainfrom
kumarUjjawal:feature/multi-location-external-table

kumarUjjawal commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026 •

edited

Loading

Uh oh!

martin-g Jun 1, 2026

Uh oh!

martin-g Jun 1, 2026

Uh oh!

martin-g Jun 1, 2026

Uh oh!

martin-g Jun 1, 2026

Uh oh!

martin-g Jun 1, 2026

Uh oh!

martin-g Jun 1, 2026

Uh oh!

kumarUjjawal commented Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kumarUjjawal commented Jun 1, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martin-g Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

martin-g Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

martin-g Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

martin-g Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

martin-g Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

martin-g Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

kumarUjjawal commented Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jun 1, 2026 •

edited

Loading