Repository navigation
[Bug] Table-model UDFs fail on TAG/ATTRIBUTE predicates, SHOW/COUNT DEVICES and UPDATE since 2.0.11 #19178
Description
Search before asking
- I searched in the issues and found nothing similar.
Version
The official apache/iotdb:2.0.11-standalone Docker image (2.0.11, build 7c0de06, linux/arm64), and master at d8800f1 built from source. The same steps work on apache/iotdb:2.0.10-standalone (build 3c26382).
Describe the bug and provide the minimal reproduce step
A user-defined table scalar function fails when it is used on a TAG or ATTRIBUTE column in WHERE, in SHOW DEVICES / COUNT DEVICES ... WHERE, or in UPDATE ... SET. The same function works in the SELECT list, and the built-in upper() works in all of these places.
The function only returns its argument:
package lazy;
import org.apache.iotdb.udf.api.customizer.analysis.ScalarFunctionAnalysis;
import org.apache.iotdb.udf.api.customizer.parameter.FunctionArguments;
import org.apache.iotdb.udf.api.relational.ScalarFunction;
import org.apache.iotdb.udf.api.relational.access.Record;
import org.apache.iotdb.udf.api.type.Type;
public class StrF implements ScalarFunction {
@Override
public ScalarFunctionAnalysis analyze(FunctionArguments arguments) {
return new ScalarFunctionAnalysis.Builder().outputDataType(Type.STRING).build();
}
@Override
public Object evaluate(Record input) {
return input.isNull(0) ? null : input.getBinary(0);
}
}CREATE DATABASE db;
CREATE TABLE db.t (device_id STRING TAG, attr1 STRING ATTRIBUTE, v INT64 FIELD);
INSERT INTO db.t(time, device_id, attr1, v) VALUES (1,'d1','a',1),(2,'d1','a',2),(1,'d2','b',100);
CREATE FUNCTION str_f AS 'lazy.StrF' USING URI 'file:/tmp/lazyudf.jar';
SELECT time, device_id, str_f(device_id) FROM db.t ORDER BY device_id, time;
SELECT count(*) FROM db.t WHERE str_f(device_id) = 'd1';
SELECT count(*) FROM db.t WHERE str_f(attr1) = 'a';
SHOW DEVICES FROM db.t WHERE str_f(device_id) = 'd1';
COUNT DEVICES FROM db.t WHERE str_f(device_id) = 'd1';
UPDATE db.t SET attr1 = str_f(device_id) WHERE device_id = 'd2';What did you expect to see?
What 2.0.10 returns: the first SELECT returns the three rows, both count(*) queries return 2, SHOW DEVICES returns d1 | a, COUNT DEVICES returns 1, and the UPDATE succeeds and sets attr1 of d2 to d2.
What did you see instead?
On 2.0.11 and master the first SELECT still works, but the rest fail:
SELECT count(*) ... WHERE str_f(device_id) = 'd1' 301: outerQueryDeadlineMs must be positive for UDF execution
SELECT count(*) ... WHERE str_f(attr1) = 'a' 301: outerQueryDeadlineMs must be positive for UDF execution
SHOW DEVICES ... WHERE str_f(device_id) = 'd1' 301: outerQueryDeadlineMs must be positive for UDF execution
COUNT DEVICES ... WHERE str_f(device_id) = 'd1' 301: IoTDBLocalFactory must not be null for UDF execution
UPDATE db.t SET attr1 = str_f(device_id) ... 305: null
For the UPDATE, the DataNode log shows IoTDBLocalFactory must not be null for UDF execution, thrown from SchemaRegionMemoryImpl#constructDevicePredicateUpdater under SchemaRegionStateMachine#write. On 2.0.11, the same statements with upper() in place of str_f all succeed.
Anything else?
The messages come from TableOperatorGenerator.IoTDBLocalFactory#createIoTDBLocal, added with IoTDBLocal in #18035. UserDefineScalarFunctionTransformer calls it in its constructor, so every user-defined scalar function needs a factory, a fragment instance id, a query id and a positive deadline, even when it never uses IoTDBLocal.
- For schema fragments,
FragmentInstanceManager#execSchemaQueryFragmentInstancecreates theFragmentInstanceContextwithout a deadline, so it stays at -1. The data path passesDriverTask.computeDeadlineTimeInMs(instance.getTimeOut()). COUNT DEVICESbuilds its transformer context inDataNodeTableOperatorGenerator#visitTableDeviceQueryCountwithout a factory.UPDATEbuilds its transformers insideSchemaRegionStateMachine#write, where there is no query.
The UPDATE case also came up in a review comment on #18889 (#18889 (comment)), and I ran into the rest while testing #19107.
I can see two directions: give schema fragments the same deadline and factory as data fragments, and create IoTDBLocal only when a function actually uses it, with a clear error where there is no query, as in UPDATE. I'm happy to send a PR once it's clear which one is preferred.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity
shuwenwei commentedon Nov 9, 2026
Thanks for the detailed analysis and reproduction. We agree that allowing UDFs in
UPDATEexposes a broader correctness problem than the missing QueryScope alone.The current schema WAL stores the raw
TableDeviceAttributeUpdateNodeand replays it by re-evaluating the UDF expression. This means:DROP FUNCTION, or if the UDF jar is missing, replay can fail;A snapshot triggered by
DROP FUNCTIONis not a complete fix. It would require a cluster-wide barrier, all affected replicas to reach a consistent snapshot/log boundary before removing the UDF or jar, and it would still not solve non-deterministic re-evaluation while the function remains registered.For the short term, we plan to reject external UDFs in
UPDATEassignments and predicates at analysis time, while keeping built-in functions and non-UDFUPDATEstatements working. This is intended as a correctness guard rather than the final design: supporting UDFs inUPDATElater would require deterministic, materialized WAL entries containing the actual device/attribute changes rather than an expression that must be evaluated again during recovery.This does not address the remaining non-
UPDATEpaths reported in #19178, such as TAG/ATTRIBUTE predicates andSHOW DEVICES/COUNT DEVICES; those need separate IoTDBLocal/context handling. #19107 will remain focused on the classloader lifetime problem.Thanks for offering the jar and script. A minimal reproducer would also be useful as a regression case for the rejection behavior.
shuwenwei commentedon Nov 9, 2026
Thanks again for the detailed report and for validating #19107.
For the non-
UPDATEcases in this issue, classloader lifetime is not the only issue. #19107 already adds a QueryScope around schema fragment planning, but these paths still have separate query-context requirements that need to be handled independently.Do you still plan to submit a separate PR for these non-
UPDATEcases?PDGGK commentedon Nov 9, 2026
Yes, it's up as #19195. Schema fragments now get the same deadline as data fragments, and
COUNT DEVICESpasses theIoTDBLocalcontext to its filter, so the TAG/ATTRIBUTE predicates andSHOW/COUNT DEVICESfrom this issue work, with or withoutIoTDBLocal.UPDATEisn't touched.For the regression case on the rejection, you only need a scalar UDF and the table from the description: the
StrFclass there, orDeviceNameFunctionfrom the IT module, which also takes and returns a string. These are the forms I'd cover. All three fail with305: nulltoday, on master and with the PR:UPDATE db.t SET attr1 = upper(device_id) WHERE device_id = 'd2'works and should keep working.DELETE DEVICESalready rejects a UDF at analysis time in all eight forms I tried, so that one seems fine.