Repository navigation
Conversation
A table scalar UDF creates its IoTDBLocal when the transformer is built, which needs a factory, a fragment instance id, a query id and a positive deadline. Schema fragments had no deadline, and the COUNT DEVICES filter got none of the four, so any UDF evaluated there failed. Give schema fragments the same deadline as data fragments, and pass the four values to the COUNT DEVICES filter.
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes the
SELECT,SHOW DEVICESandCOUNT DEVICEScases of #19178. TheUPDATEcase is left out on purpose, see below.Since 2.0.11, a table scalar UDF fails when it runs in a schema fragment, which is what happens when it is applied to a TAG or ATTRIBUTE column in
WHERE. It fails even if the function never usesIoTDBLocal. With a UDF that just returns its argument, and a table with devicesd1(attr1 = 'a', 2 rows) andd2(attr1 = 'b', 1 row):SELECT count(*) FROM db.t WHERE str_f(device_id) = 'd1'301: outerQueryDeadlineMs must be positive for UDF executionSELECT count(*) FROM db.t WHERE str_f(attr1) = 'a'SHOW DEVICES FROM db.t WHERE str_f(device_id) = 'd1'd1,aCOUNT DEVICES FROM db.t WHERE str_f(device_id) = 'd1'301: IoTDBLocalFactory must not be null for UDF executionCause
UserDefineScalarFunctionTransformercreates itsIoTDBLocalin the constructor, andIoTDBLocalFactory#createIoTDBLocalneeds a factory, a fragment instance id, a query id and a positive deadline. Two places do not supply them:FragmentInstanceManager#execSchemaQueryFragmentInstancecreates the context without a deadline, so it stays at -1. The data path passesDriverTask.computeDeadlineTimeInMs(instance.getTimeOut()).DataNodeTableOperatorGenerator#visitTableDeviceQueryCountbuilds itsColumnTransformerBuilder.Contextwith the shorter constructor, which leaves all four unset.SHOW DEVICESdoes not have this problem because its filter goes through the normal filter operator.Change
COUNT DEVICESpasses the four values to its context.Nothing else reads the deadline, so schema queries without a UDF behave as before. A UDF that does call
IoTDBLocalnow also works in these predicates, and its nested query runs as the current user, because the device fetch carries the outer query's session.Not in this PR
UPDATEwith a UDF still fails. The plan in [Bug] Table-model UDFs fail on TAG/ATTRIBUTE predicates, SHOW/COUNT DEVICES and UPDATE since 2.0.11 #19178 is to reject it at analysis time.createFragmentInstanceContextcall inexecSchemaQueryFragmentInstance, so whichever of the two goes in second needs a small rebase.COUNT DEVICES FROM view WHERE upper(device_id) = 'D1'returns 0 on master, whileSHOW DEVICESwith the same condition finds the device. A UDF there no longer fails with this PR, but it gets the same 0. That comes from howDevicePredicateHandlerpicks the tag position, it doesn't depend on UDFs, so I left it out.Verification
cd31721against this change. The statements above andSELECT ... WHERE str_f(device_id) = 'd2' AND v > 1, each run three times: right after the insert, again with the device cache warm, and afterCLEAR ATTRIBUTE CACHE. Master fails every time and this change returns the right result every time.SELECT count(*)andSHOW DEVICESwith the UDF on a table view also return the right result.IoTDBLocalIT: a UDF that does not useIoTDBLocal, a UDF that queries throughIoTDBLocal, and a user without permission on the table the UDF reads, who getsAccess DeniedfromSELECT,SHOW DEVICESandCOUNT DEVICES.-PTableSimpleIT(23 tests, one existing test is skipped there).-PTableClusterIT(1C3D). The three new tests passed every time. One run passed all 23; the other two hit the timeouts described below.outerQueryDeadlineMs must be positive. With only theCOUNT DEVICESchange they fail the same way. With only the deadline change they fail atCOUNT DEVICESwithIoTDBLocalFactory must not be null.spotless:checkpasses for datanode and integration-test.Cluster timeouts in
IoTDBLocalITThis is not caused by this PR, but the cluster job may show it. With
-PTableClusterIT, nine existing tests that run nestedIoTDBLocalqueries (testDeviceNameWithSingleQuery,testLocalQueryUdafAtBeforeStartand others) can time out after 120 s. I saw the same nine on mastercd31721without this change, and in two of the three runs above. A thread dump taken during the timeout, on master, shows all sixQuery-Workerthreads of one DataNode blocked inQueryExecution.getBatchResult, inside nested queries or the device fetch of a nested query. That looks like the worker pool running out, since those queries need worker threads too. I can open a separate issue with the dump if that helps.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
FragmentInstanceManager,FragmentInstanceContextDataNodeTableOperatorGeneratorIoTDBLocalIT