Skip to content

fix: Use thread context classloader for Iceberg class loading in CometScanRule#4609

Merged
mbutrovich merged 2 commits into
apache:mainfrom
chern:classloader-fix
Jun 9, 2026
Merged

fix: Use thread context classloader for Iceberg class loading in CometScanRule#4609
mbutrovich merged 2 commits into
apache:mainfrom
chern:classloader-fix

Conversation

@chern

@chern chern commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #3737.

Rationale for this change

Similar change to #3738.

Comet uses Class.forName(className) to reflectively load Iceberg classes.
The single-argument Class.forName uses the classloader of the calling class, not the thread context classloader.

When the Comet JAR is placed in Spark's jars/ folder, it is loaded by the system classloader.
However, Iceberg classes may be bundled inside the user's application jar (as uber jar), which is loaded by Spark's application classloader .
Since parent classloaders cannot see classes loaded by child classloaders, Class.forName from Comet's code fails with ClassNotFoundException.

What changes are included in this PR?

  • Replaces bare Class.forName() calls in CometScanRule with the IcebergReflection.loadClass() helper method.

How are these changes tested?

  • Verified in a deployment where the Comet jar is in Spark's jars/ folder and Iceberg is bundled in the application's uber jar; ran a few SQL queries that involved reading from Iceberg.

@mbutrovich mbutrovich self-requested a review June 8, 2026 20:02
@chern chern changed the title Use thread context classloader for Iceberg class loading in CometScanRule fix: Use thread context classloader for Iceberg class loading in CometScanRule Jun 8, 2026
@karuppayya

Copy link
Copy Markdown
Contributor

@mbutrovich Do we need a sperate issue tracking this or reuse old?

@mbutrovich

Copy link
Copy Markdown
Contributor

@mbutrovich Do we need a sperate issue tracking this or reuse old?

No preference.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @chern!

@mbutrovich mbutrovich merged commit ec9a625 into apache:main Jun 9, 2026
@chern chern deleted the classloader-fix branch June 9, 2026 16:15
@karuppayya

Copy link
Copy Markdown
Contributor

@mbutrovich @andygrove Can you please let us know when the next release is planned?

@mbutrovich

Copy link
Copy Markdown
Contributor

@mbutrovich @andygrove Can you please let us know when the next release is planned?

Discussion is ongoing here #4564.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iceberg class loading fails with ClassNotFoundException when Comet JAR is on system classpath

3 participants