Skip to content

[Console] Fix four DTO/entity defects that break Spark and Flink SQL app management - #4497

Open
88fantasy wants to merge 1 commit into
apache:devfrom
88fantasy:fix/console-spark-dto-defects
Open

[Console] Fix four DTO/entity defects that break Spark and Flink SQL app management#4497
88fantasy wants to merge 1 commit into
apache:devfrom
88fantasy:fix/console-spark-dto-defects

Conversation

@88fantasy

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Four small DTO/entity defects, each of which makes a normal UI operation fail outright on a clean install. They are unrelated to each other in cause but identical in shape — a field that does not line up between the request DTO, the entity, and the column — so they are grouped in one PR.

Closes #4491
Closes #4492
Closes #4493
Closes #4494

Brief change log

  1. FlinkAppCreateRequest declares sqlId ([Bug] Saving an already-started Flink SQL application from the UI always fails (FlinkAppCreateRequest has no sqlId) #4491). The frontend sends it and updateFlinkSqlJob() requires it, but the DTO had no such field, so BeanUtils.copyProperties skipped it and saving any already-started Flink SQL application failed with Flink sql is null, update flink sql job failed. SparkAppCreateRequest and FlinkAppResponse already declare it — this was a one-sided omission.

  2. SparkAppStateEnum.of(Integer) returns OTHER for null ([Bug] Spark application list page returns 500 (SparkAppStateEnum.of unboxes a null state) #4492). It compared with appState.value == state, which unboxes; SparkAppListQueryRequest's scalar state is null for an unfiltered list, so /spark/app/list returned 500 for everyone. FlinkAppStateEnum.getState(Integer) has the same unboxing shape but nothing reaches it with a null today, so it is deliberately left alone here.

  3. SparkEnv.doSetSparkConf() stores an empty conf when the file is absent ([Bug] A stock Spark distribution cannot be registered (spark-defaults.conf is absent, spark_conf is NOT NULL) #4493). A stock Spark distribution ships only spark-defaults.conf.template, so sparkConf stayed null and the insert failed on t_spark_env.spark_conf (NOT NULL, no default) — no Spark home could be registered at all.

  4. SparkApplication.k8sImagePullPolicy is Integer ([Bug] Creating a Spark application from the UI always returns 500 (k8sImagePullPolicy primitive int) #4494), matching its nullable column and the Integer in SparkAppCreateRequest. As a primitive it unboxed the null the Spark frontend never sends, so /spark/app/create returned 500 for every UI-created application.

Verifying this change

  • (1) and (2) were verified against a running console: the same update request goes from always-failing to succeeding, and /spark/app/list from 500 to 200.
  • (3) was verified by the inverse — creating an empty conf/spark-defaults.conf in an otherwise stock Spark home makes registration succeed with the file's existence as the only changed variable.
  • mvn test on streampark-console-service passes (93 tests, 0 failures).

Does this pull request potentially affect one of the following parts

  • Dependencies (does it add or upgrade a dependency): no
  • The public API: yes, minorFlinkAppCreateRequest gains an optional sqlId, and SparkApplication.getK8sImagePullPolicy() now returns Integer rather than int.
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment: no

Documentation

  • Does this pull request introduce a new feature? no

…app management

Each of these makes a normal UI operation fail outright; all four are
reproducible on a clean install.

1. FlinkAppCreateRequest carries no sqlId, though the frontend sends one
   and FlinkApplicationManageServiceImpl.updateFlinkSqlJob() requires it
   for an application that has already been started. BeanUtils.copyProperties
   silently skips the absent field, the entity's sqlId stays null, and the
   save fails with "Flink sql is null, update flink sql job failed." Only
   never-started applications were unaffected, since those take the
   candidate branch. SparkAppCreateRequest and FlinkAppResponse both
   already declare sqlId — this was a one-sided omission on the Flink
   request DTOs.

2. SparkAppStateEnum.of(Integer) compares appState.value == state, which
   unboxes. SparkAppListQueryRequest has a scalar state field that is null
   whenever the list is not filtered by status, so the copy to
   SparkApplication leaves state null and shouldTracking() -> getStateEnum()
   -> of(null) throws NPE: /spark/app/list returns 500 for everyone. Now
   returns OTHER for null. FlinkAppStateEnum.getState(Integer) has the same
   unboxing shape but no scalar state field reaching it today, so it is
   left alone here.

3. SparkEnv.doSetSparkConf() only assigned sparkConf when conf/spark-defaults.conf
   existed. A stock Spark distribution ships only spark-defaults.conf.template,
   so for an unmodified Spark home the field stays null and the insert fails
   on t_spark_env.spark_conf, which is NOT NULL with no default — no Spark
   home could be registered at all. An absent file is now stored as an empty
   conf.

4. SparkApplication.k8sImagePullPolicy was a primitive int while
   SparkAppCreateRequest declares Integer and the Spark frontend never sends
   the field, so BeanUtils.copyProperties unboxed null and /spark/app/create
   returned 500 for every UI-created Spark application. The entity field now
   matches its nullable column.
@sonarqubecloud

Copy link
Copy Markdown

@wolfboys

Copy link
Copy Markdown
Member

I noticed your recent code contributions — they look good. I wanted to ask which AI model you used, including the specific model/version, and how you prompted the model to complete this contribution.

We’re currently planning to introduce some guidelines and best practices for AI-assisted coding in our open-source project, so your experience would be very helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment