[Flink] Fix Flink SQL job submission failures on the client path - #4496
Open
88fantasy wants to merge 1 commit into
Open
[Flink] Fix Flink SQL job submission failures on the client path#449688fantasy wants to merge 1 commit into
88fantasy wants to merge 1 commit into
Conversation
Six defects on the path that turns a saved Flink SQL application into a
submitted job. Each was found by submitting a real Flink SQL job to a
standalone cluster and fixing whatever failed next; they are independent
of each other but all sit on this one path.
1. ClassLoaderUtils.runAsClassLoader restored the context classloader
captured in a static field when the class was first initialized —
whichever thread happened to load it — instead of the one the calling
thread had on entry. On pooled threads that silently replaces an
unrelated thread's context classloader.
2. FlinkClientTrait.getCustomCommandLines and
RemoteClient.getStandAloneClusterDescriptor call into Flink classes
bound to the Flink version bundled with this module, while the calling
thread's context classloader is FlinkShimsProxy's target-version shims
classloader. Their internal ServiceLoader lookups therefore resolved
providers from a different Flink version than the interfaces bundled
here, failing with ServiceConfigurationError ("not a subtype"). Both
call sites now run under their own class's classloader. Closes apache#4483.
3. The build-response getters (workspacePath, pass, shadedJarPath,
flinkBaseImage, mainJarPath, extraLibJarPaths, flinkImageTag,
podTemplatePaths, dockerInnerMainJarPath) do not follow JavaBean
getter naming and carried no @JsonProperty, so Jackson silently
skipped them: every build result persisted to t_flink_app's
buildResultJson lost its paths, and only pass survived — by the
coincidence that its field default is already true. A later submit
then read back shadedJarPath == null and failed with an NPE, an
"entry point class not found", or "flinkJobJar is null", depending on
which downstream path consumed it.
4. SubmitRequest.userJarFile() passed shadedJarPath() straight to
new File(...), which throws NPE when it is legitimately null.
5. streampark-console-service declared a compile dependency on
streampark-flink-shims-base but not on streampark-flink-shims-base-v2,
so FlinkTableInitializerV2 never reached the console's lib/ and every
Flink 2.x SQL job failed with NoClassDefFoundError. Flink 1.x was
unaffected, which is why this went unnoticed.
6. PackagedProgram's setUserClassPaths, disabled wholesale for apache#3761, is
re-enabled for FLINK_SQL jobs only, so a SQL job's connector jars reach
the client classpath. Verified against a real cluster not to reproduce
the ClassCastException apache#3761 describes, and it leaves every other job
type on the existing behaviour.
|
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Six defects on the path that turns a saved Flink SQL application into a submitted job. They are independent of each other, but each one only becomes reachable after the previous is fixed, which is why they are together here — each was found by submitting a real Flink SQL job to a standalone cluster and fixing whatever failed next.
Closes #4483
Closes #4488
Closes #4489
Closes #4490
Brief change log
ClassLoaderUtils.runAsClassLoaderrestores the caller's own context classloader ([Bug] ClassLoaderUtils.runAsClassLoader restores a stale context classloader captured at class-init time #4489). It previously restored astatic finalfield captured at class-initialisation time — whichever thread first loaded the class. On pooled threads that silently installs an unrelated classloader and leaves it there.ORIGINAL_CLASS_LOADERstays, sincecloneClassLoader()still uses it.FlinkClientTrait.getCustomCommandLinesandRemoteClient.getStandAloneClusterDescriptorrun under their own class's classloader ([Bug] PipelineExecutorFactory ServiceConfigurationError when submitting a Flink job (ChildFirstClassLoader / console baseline Flink version conflict) #4483). Both call Flink classes bound to the Flink version bundled with this module, while the calling thread's context classloader isFlinkShimsProxy's target-version shims classloader. Their internalServiceLoaderlookups therefore resolved providers from a different Flink version than the interfaces bundled here, throwingServiceConfigurationError: ... not a subtype.Build-response getters carry
@JsonProperty([Bug] Build result paths are silently dropped when persisted, breaking every subsequent job start #4488). They do not follow JavaBean naming, so Jackson silently omitted them: every persisted build result lostshadedJarPath/workspacePath, and onlypassappeared to survive — by the coincidence that its field default is alreadytrue. Method names are unchanged, so no caller is affected. The K8s and Docker build responses had the same defect and are fixed with it.SubmitRequest.userJarFile()tolerates a nullshadedJarPathinstead of passing it tonew File(...).streampark-console-servicedepends onstreampark-flink-shims-base-v2([Bug] All Flink 2.x SQL jobs fail with NoClassDefFoundError: FlinkTableInitializerV2 #4490), soFlinkTableInitializerV2reaches the console'slib/. Only the v1 module was declared, so every Flink 2.x SQL job failed withNoClassDefFoundError.setUserClassPathsis re-enabled forFLINK_SQLjobs only. It was disabled wholesale for [Bug] Demo Flink SQL can not be run on yarn-per-job mode with flink 1.16 and flink 1.17 #3761, whose reported scenario is a different deploy mode; a SQL job needs its connector jars on the client classpath. Every other job type keeps the current behaviour.Verifying this change
ShadedBuildResponseserialisation was checked by round-tripping it through a plainObjectMapperinjshellagainst the built jar —shadedJarPathis absent from the JSON before the change and present after.mvn installoverstreampark-common,streampark-flink-client-api,streampark-flink-client-coreandstreampark-flink-packerpasses, spotless/checkstyle/scalastyle included.Does this pull request potentially affect one of the following parts
streampark-flink-shims-base-v2is now packaged into the console distribution.Documentation