Skip to content

Fix table-model UDFs on TAG/ATTRIBUTE predicates and SHOW/COUNT DEVICES - #19195

Open
PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:fix-table-udf-schema-fragment-context
Open

PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:fix-table-udf-schema-fragment-context

Conversation

@PDGGK

@PDGGK PDGGK commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes the SELECT, SHOW DEVICES and COUNT DEVICES cases of #19178. The UPDATE case is left out on purpose, see below.

Since 2.0.11, a table scalar UDF fails when it runs in a schema fragment, which is what happens when it is applied to a TAG or ATTRIBUTE column in WHERE. It fails even if the function never uses IoTDBLocal. With a UDF that just returns its argument, and a table with devices d1 (attr1 = 'a', 2 rows) and d2 (attr1 = 'b', 1 row):

statement master this PR
SELECT count(*) FROM db.t WHERE str_f(device_id) = 'd1' 301: outerQueryDeadlineMs must be positive for UDF execution 2
SELECT count(*) FROM db.t WHERE str_f(attr1) = 'a' same error 2
SHOW DEVICES FROM db.t WHERE str_f(device_id) = 'd1' same error d1, a
COUNT DEVICES FROM db.t WHERE str_f(device_id) = 'd1' 301: IoTDBLocalFactory must not be null for UDF execution 1

Cause

UserDefineScalarFunctionTransformer creates its IoTDBLocal in the constructor, and IoTDBLocalFactory#createIoTDBLocal needs a factory, a fragment instance id, a query id and a positive deadline. Two places do not supply them:

  • FragmentInstanceManager#execSchemaQueryFragmentInstance creates the context without a deadline, so it stays at -1. The data path passes DriverTask.computeDeadlineTimeInMs(instance.getTimeOut()).
  • DataNodeTableOperatorGenerator#visitTableDeviceQueryCount builds its ColumnTransformerBuilder.Context with the shorter constructor, which leaves all four unset. SHOW DEVICES does not have this problem because its filter goes through the normal filter operator.

Change

  • Schema fragments get a deadline computed the same way as data fragments.
  • COUNT DEVICES passes the four values to its context.

Nothing else reads the deadline, so schema queries without a UDF behave as before. A UDF that does call IoTDBLocal now also works in these predicates, and its nested query runs as the current user, because the device fetch carries the outer query's session.

Not in this PR

Verification

  • Real server, master cd31721 against this change. The statements above and SELECT ... WHERE str_f(device_id) = 'd2' AND v > 1, each run three times: right after the insert, again with the device cache warm, and after CLEAR ATTRIBUTE CACHE. Master fails every time and this change returns the right result every time. SELECT count(*) and SHOW DEVICES with the UDF on a table view also return the right result.
  • New tests in IoTDBLocalIT: a UDF that does not use IoTDBLocal, a UDF that queries through IoTDBLocal, and a user without permission on the table the UDF reads, who gets Access Denied from SELECT, SHOW DEVICES and COUNT DEVICES.
    • The class passes with -PTableSimpleIT (23 tests, one existing test is skipped there).
    • I ran it three times with -PTableClusterIT (1C3D). The three new tests passed every time. One run passed all 23; the other two hit the timeouts described below.
    • On master all three new tests fail with outerQueryDeadlineMs must be positive. With only the COUNT DEVICES change they fail the same way. With only the deadline change they fail at COUNT DEVICES with IoTDBLocalFactory must not be null.
  • spotless:check passes for datanode and integration-test.

Cluster timeouts in IoTDBLocalIT

This is not caused by this PR, but the cluster job may show it. With -PTableClusterIT, nine existing tests that run nested IoTDBLocal queries (testDeviceNameWithSingleQuery, testLocalQueryUdafAtBeforeStart and others) can time out after 120 s. I saw the same nine on master cd31721 without this change, and in two of the three runs above. A thread dump taken during the timeout, on master, shows all six Query-Worker threads of one DataNode blocked in QueryExecution.getBatchResult, inside nested queries or the device fetch of a nested query. That looks like the worker pool running out, since those queries need worker threads too. I can open a separate issue with the dump if that helps.


This PR has:

  • been self-reviewed.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • FragmentInstanceManager, FragmentInstanceContext
  • DataNodeTableOperatorGenerator
  • IoTDBLocalIT

A table scalar UDF creates its IoTDBLocal when the transformer is built,
which needs a factory, a fragment instance id, a query id and a positive
deadline. Schema fragments had no deadline, and the COUNT DEVICES filter
got none of the four, so any UDF evaluated there failed.

Give schema fragments the same deadline as data fragments, and pass the
four values to the COUNT DEVICES filter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant