fix(build): bundle hive-exec in the utilities bundle under the shade-hive profile - #19800
Open
rangareddy wants to merge 1 commit into
Open
fix(build): bundle hive-exec in the utilities bundle under the shade-hive profile#19800rangareddy wants to merge 1 commit into
rangareddy wants to merge 1 commit into
Conversation
…hive profile Closes apache#16374 (HUDI-7321). Building the utilities bundle with -Putilities-bundle-shade-hive produced a jar that fails at runtime with: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat hudi-hadoop-mr is in the bundle's shade whitelist, and its HoodieParquetInputFormatBase extends MapredParquetInputFormat, which lives in hive-exec. The bundle declared five hive artifacts - hive-service, hive-service-rpc, hive-jdbc, hive-metastore, hive-common - and whitelisted the same five. hive-exec was in neither list, and its managed scope is provided, which never propagates transitively, so nothing could pull it in. The bundle therefore shipped subclasses whose superclass was absent. Declares hive-exec at ${hive.exec.classifier} (core) with scope ${utilities.bundle.hive.scope}, matching the hive artifacts alongside it, and adds it to the shade whitelist. The core classifier carries Hive's own classes without Hive's shaded copies of third-party libraries. Scope defaults to provided, so the default bundle is unchanged and hive-exec is packaged only under -Putilities-bundle-shade-hive. No local <exclusions> on the new dependency, deliberately: the root pom's dependencyManagement entry for hive-exec already excludes javax.mail, the jetty aggregate, pentaho, log4j 1.x, log4j2, slf4j-log4j12 and hbase, several of which the enforcer bans. Declaring exclusions locally replaces that managed set rather than adding to it. Verified by building the bundle both ways and loading the class from the jar with a Hadoop-only classpath, which is what a cluster running this bundle provides: before, -Putilities-bundle-shade-hive MapredParquetInputFormat -> ClassNotFoundException HoodieParquetInputFormat -> NoClassDefFoundError: org/apache/hadoop/hive/ql/io/parquet/ MapredParquetInputFormat after, -Putilities-bundle-shade-hive MapredParquetInputFormat -> resolves HoodieParquetInputFormat -> resolves Jar contents: MapredParquetInputFormat goes from 0 to 1 entry under the profile and stays at 0 without it. Bundle grows 101.4 MB -> 111.6 MB under the profile; the default bundle is 90.4 MB with zero hive-exec classes. Full build with checkstyle, RAT and enforcer enabled passes, and the shaded jar contains nothing under org/apache/log4j, org/apache/logging/log4j, org/slf4j/impl, org/apache/hadoop/hbase, org/pentaho or javax/mail. Not changed: hudi-utilities-slim-bundle. It ships only hudi-utilities among the hudi artifacts, not hudi-hadoop-mr, so it has no dangling superclass to fix, and bundling 10 MB of hive-exec would defeat the point of a slim bundle. Its utilities-bundle-shade-hive profile is however dead code - it sets utilities.bundle.hive.scope, which that pom never references, and its hive relocations are inert because no hive artifact is ever packaged. Worth a follow-up.
rangareddy
marked this pull request as ready for review
August 31, 2026 09:54
Collaborator
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19800 +/- ##
============================================
- Coverage 78.26% 75.88% -2.39%
+ Complexity 33806 32864 -942
============================================
Files 2541 2541
Lines 141586 141676 +90
Branches 17142 17253 +111
============================================
- Hits 110815 107504 -3311
- Misses 23052 26477 +3425
+ Partials 7719 7695 -24
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
hudi-agent
reviewed
Sep 1, 2026
hudi-agent
left a comment
hudi-agent
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the contribution! This PR adds hive-exec (core classifier) to the utilities bundle's dependency list and shade whitelist so the bundled hudi-hadoop-mr input formats ship with their MapredParquetInputFormat superclass under the -Putilities-bundle-shade-hive profile. The change mirrors how every other module and bundle declares hive-exec (same classifier and scope property), the provided default keeps the standard bundle unchanged, and relying on the root pom's managed exclusions rather than local ones is the correct call. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.
cc @yihua
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Closes #16374 (JIRA: HUDI-7321).
Building the utilities bundle with
-Putilities-bundle-shade-hiveproduces a jar that fails at runtime with:Root cause.
hudi-hadoop-mris in the bundle's shade whitelist(
packaging/hudi-utilities-bundle/pom.xml:106), andHoodieParquetInputFormatBase:49 extends MapredParquetInputFormat, which lives inhive-exec. The bundle declared five hive artifacts -hive-service,hive-service-rpc,hive-jdbc,hive-metastore,hive-common- and whitelisted the samefive.
hive-execwas in neither list, and its managed scope isprovided, which never propagatestransitively, so nothing could pull it in. The bundle therefore shipped subclasses whose superclass was
absent.
The reporter's workaround, "adding hive-exec resolved it", is the correct diagnosis.
Summary and Changelog
One file,
packaging/hudi-utilities-bundle/pom.xml, +27:hive-execat${hive.exec.classifier}withscope=${utilities.bundle.hive.scope}, matching thehive artifacts alongside it. The
coreclassifier is what the rest of the build uses: it carries Hive's ownclasses without Hive's shaded copies of third-party libraries, so it does not drag protobuf, guava and
friends into the bundle (10.3 MB rather than 44.2 MB).
<include>org.apache.hive:hive-exec</include>to the shade whitelist.Scope defaults to
provided(root pom), so the default bundle is unchanged andhive-execis packagedonly under
-Putilities-bundle-shade-hive.The new dependency deliberately carries no local
<exclusions>. The root pom'sdependencyManagemententry for
hive-execalready excludesjavax.mail, the jetty aggregate, pentaho, log4j 1.x, log4j2,slf4j-log4j12and hbase, several of which the enforcer bans outright. Declaring exclusions locallyreplaces that managed set rather than adding to it, so an innocuous-looking local exclusion block would
quietly let the banned artifacts back in.
Verification
Built the bundle both ways, then loaded the class from the jar with a Hadoop-only classpath - which is
what a cluster running this bundle provides, and which is why the bundle has to carry hive itself:
MapredParquetInputFormatHoodieParquetInputFormat-Putilities-bundle-shade-hiveClassNotFoundExceptionNoClassDefFoundError: org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat-Putilities-bundle-shade-hiveThe "before" row is the reported failure, reproduced.
Jar contents and sizes:
MapredParquetInputFormatlands under its original name; nothing is relocated underorg.apache.hudi.(0 entries under
org/apache/hudi/org/apache/hadoop/hive/ql/), so Hive's own loading ofHoodieParquetInputFormatis unaffected.Full build with checkstyle, RAT and enforcer enabled passes (
BannedDependencies passed), and the shadedjar contains nothing under
org/apache/log4j,org/apache/logging/log4j,org/slf4j/impl,org/apache/hadoop/hbase,org/pentahoorjavax/mail.Impact
Only the opt-in
-Putilities-bundle-shade-hivebuild changes: that bundle grows by ~10 MB and becomesself-contained for the Hive input formats it already ships. The default
hudi-utilities-bundleis byte-for-byteunaffected, since
providedscope keepshive-execout and the whitelist entry has nothing to match.Because
org.apache.hadoop.hive.ql.is not in this bundle's relocation list, the profile's bundle now carriesHive
qlclasses under their original names. That is the existing semantics of this profile - it exists to putHive inside the bundle - but it does mean the shaded jar can shadow a Hive install on a shared classpath.
Relocating
hive.qlinstead was considered and rejected: it would rewriteHoodieParquetInputFormatBase'ssuperclass and break Hive's own loading of
HoodieParquetInputFormat.Risk Level
low - opt-in profile only, default bundle unchanged, verified against the enforcer.
Documentation Update
none
Contributor's checklist