[Bug]: Production system broken for >= 9.1.13

[arrabito]

Search before creating an issue

• I have searched existing issues and confirmed this is not a duplicate

Bug Description

The PS is broken for 9.1.16, while it's working for 9.1.12. I suspect that it's broken for >= 9.1.13 due to the changes here:
18774a9

The production can be created correctly and the corresponding transformations are also correctly created, but it seems that the format of the transformation body is not correct, so that the WorkflowTaskAgent gives error when trying to submit tasks:

2026-09-03T12:38:09,946990Z Transformation/WorkflowTaskAgent/[2] .submitTasks [140280454964800] INFO: Obtained 3 tasks for submission
2026-09-03T12:38:09,957319Z Transformation/WorkflowTaskAgent/[2] ._execute [140280454964800] ERROR: Exception executing operation submitTasks
Traceback (most recent call last):
  File "/opt/dirac/diracos/lib/python3.12/site-packages/DIRAC/TransformationSystem/Agent/TaskManagerAgentBase.py", line 293, in _execute
    res = getattr(self, operation)(transDict, clients)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/site-packages/DIRAC/TransformationSystem/Agent/TaskManagerAgentBase.py", line 593, in submitTasks
    res = self._prepareAndSubmitAndUpdateTasks(transID, transBody, taskDictChunk, owner, ownerGroup, clients)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/site-packages/DIRAC/TransformationSystem/Agent/TaskManagerAgentBase.py", line 615, in _prepareAndSubmitAndUpdateTasks
    preparedTransformationTasks = clients["TaskManager"].prepareTransformationTasks(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/site-packages/DIRAC/TransformationSystem/Client/WorkflowTasks.py", line 111, in prepareTransformationTasks
    return self._prepareTasksBulk(transBody, taskDict, owner, ownerGroup)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/site-packages/DIRAC/TransformationSystem/Client/WorkflowTasks.py", line 134, in _prepareTasksBulk
    oJob = self.jobClass(transBody)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/site-packages/DIRAC/Interfaces/API/Job.py", line 92, in __init__
    self.workflow = Workflow(script)
                    ^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/site-packages/DIRAC/Core/Workflow/Workflow.py", line 44, in __init__
    fromXMLFile(obj, self)
  File "/opt/dirac/diracos/lib/python3.12/site-packages/DIRAC/Core/Workflow/Workflow.py", line 415, in fromXMLFile
    xml.sax.parse(xml_file, handler)  # nosec: B317
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/xml/sax/__init__.py", line 33, in parse
    parser.parse(source)
  File "/opt/dirac/diracos/lib/python3.12/xml/sax/expatreader.py", line 99, in parse
    source = saxutils.prepare_input_source(source)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/xml/sax/saxutils.py", line 365, in prepare_input_source
    f = urllib.request.urlopen(source.getSystemId())
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/urllib/request.py", line 215, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/urllib/request.py", line 499, in open
    req = Request(fullurl, data)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/urllib/request.py", line 318, in __init__
    self.full_url = url
    ^^^^^^^^^^^^^
  File "/opt/dirac/diracos/lib/python3.12/urllib/request.py", line 344, in full_url
    self._parse()
  File "/opt/dirac/diracos/lib/python3.12/urllib/request.py", line 373, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: '"<Workflow>\\n<descr_short></descr_short>\\n<description><![CDATA[]]></description>\\n<name>data_generation</name>\\n<origin></origin>\\n<type></type>\\n<version>0.0</version>\\n<Parameter name=\\"JobType\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Job Type\\"><value><![CDATA[User]]></value></Parameter>\\n<Parameter name=\\"Priority\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"User Job Priority\\"><value><![CDATA[1]]></value></Parameter>\\n<Parameter name=\\"JobGroup\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Name of the JobGroup\\"><value><![CDATA[ctao.dpps.test]]></value></Parameter>\\n<Parameter name=\\"JobName\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"User specified name\\"><value><![CDATA[data_generation]]></value></Parameter>\\n<Parameter name=\\"StdOutput\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Standard output file\\"><value><![CDATA[std.out]]></value></Parameter>\\n<Parameter name=\\"StdError\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Standard error file\\"><value><![CDATA[std.err]]></value></Parameter>\\n<Parameter name=\\"InputData\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Default null input data value\\"><value><![CDATA[]]></value></Parameter>\\n<Parameter name=\\"LogLevel\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Job Logging Level\\"><value><![CDATA[INFO]]></value></Parameter>\\n<Parameter name=\\"arguments\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Arguments to executable Step\\"><value><![CDATA[]]></value></Parameter>\\n<Parameter name=\\"ParametricInputData\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Default null parametric input data value\\"><value><![CDATA[]]></value></Parameter>\\n<Parameter name=\\"ParametricInputSandbox\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Default null parametric input sandbox value\\"><value><![CDATA[]]></value></Parameter>\\n<Parameter name=\\"Site\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"User specified destination site\\"><value><![CDATA[CTAO.CI.de]]></value></Parameter>\\n<Parameter name=\\"OutputSandbox\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Output sandbox file list\\"><value><![CDATA[*log]]></value></Parameter>\\n<Parameter name=\\"OutputData\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Output data file\\"><value><![CDATA[LFN:/ctao.dpps.test/tests/productions/random_gen/data/data_%(JOB_ID)s.txt]]></value></Parameter>\\n<Parameter name=\\"OutputSE\\" type=\\"JDL\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"User specified Output SE\\"><value><![CDATA[TestSE]]></value></Parameter>\\n<ModuleDefinition>\\n<body><![CDATA[\\nfrom DIRAC.Workflow.Modules.Script import Script\\n]]></body>\\n<descr_short></descr_short>\\n<description><![CDATA[ The Script class provides a simple way for users to specify an executable\\n    or file to run (and is also a simple example of a workflow module).\\n]]></description>\\n<origin></origin>\\n<required></required>\\n<type>Script</type>\\n<version>0.0</version>\\n</ModuleDefinition>\\n<StepDefinition>\\n<descr_short></descr_short>\\n<description><![CDATA[]]></description>\\n<origin></origin>\\n<type>ScriptStep1</type>\\n<version>0.0</version>\\n<Parameter name=\\"executable\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Executable Script\\"><value><![CDATA[]]></value></Parameter>\\n<Parameter name=\\"arguments\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Arguments for executable Script\\"><value><![CDATA[]]></value></Parameter>\\n<Parameter name=\\"applicationLog\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Log file name\\"><value><![CDATA[]]></value></Parameter>\\n<ModuleInstance>\\n<descr_short></descr_short>\\n<name>Script</name>\\n<type>Script</type>\\n</ModuleInstance>\\n</StepDefinition>\\n<StepInstance>\\n<descr_short></descr_short>\\n<name>RunScriptStep1</name>\\n<type>ScriptStep1</type>\\n<Parameter name=\\"executable\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Executable Script\\"><value><![CDATA[python]]></value></Parameter>\\n<Parameter name=\\"arguments\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Arguments for executable Script\\"><value><![CDATA[-c \'\\"\\"\\"Random data generator.\\"\\"\\"\\n\\nimport argparse\\nimport random\\n\\n\\ndef generate_random_data(file_path: str = \\"data.txt\\", num_lines: int = 100):\\n    \\"\\"\\"Generate random data\\"\\"\\"\\n    with open(file_path, \\"w\\", encoding=\\"utf-8\\") as f:\\n        mu = random.randint(1, 10)\\n        sig = random.randint(1, 5)\\n        for _ in range(num_lines):\\n            rd = random.gauss(mu, sig)\\n            f.write(f\\"{rd}\\\\n\\")\\n    print(f\\"data file: {file_path}, mean: {mu}, std dev: {sig}\\")\\n\\n\\nif __name__ == \\"__main__\\":\\n    parser = argparse.ArgumentParser(description=\\"Generate random data\\")\\n    parser.add_argument(\\"--file-path\\", default=\\"data.txt\\", help=\\"Output file path\\")\\n    parser.add_argument(\\n        \\"--num_lines\\", type=int, default=100, help=\\"Number of lines to generate\\"\\n    )\\n    args = parser.parse_args()\\n    generate_random_data(args.file_path, args.num_lines)\\n\' --file-path data_@{JOB_ID}.txt]]></value></Parameter>\\n<Parameter name=\\"applicationLog\\" type=\\"string\\" linked_module=\\"\\" linked_parameter=\\"\\" in=\\"True\\" out=\\"False\\" description=\\"Log file name\\"><value><![CDATA[random_data_generation_@{JOB_ID}.log]]></value></Parameter>\\n</StepInstance>\\n</Workflow>\\n"'

Steps to Reproduce

No response

Expected Behavior

No response

Actual Behavior

No response

Environment

No response

Relevant Log Output

Additional Context

No response

[arrabito]

I think that it's due to the presence of the " character here:

'"<Workflow>

so that it's not interpreted as a string here:

https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/Core/Workflow/Workflow.py#L41

[arrabito]

I've just checked that removing the leading/ending "" character is not enough.

Indeed even if after this change, the correct fromXMLString method is called instead of fromXMLFile, the string has still not the correct xml format (see the log in my previous comment).

I've just checked that replacing:

https://github.com/DIRACGrid/DIRAC/blob/v9.1.16/src/DIRAC/ProductionSystem/DB/ProductionDB.py#L253C1-L270C60

with the previous syntax:

https://github.com/DIRACGrid/DIRAC/blob/v9.1.12/src/DIRAC/ProductionSystem/DB/ProductionDB.py#L252C1-L271C49

works fine.

Do want me to create a PR to revert to the previous syntax or do you suggest a better fix?

/cc @sfayer

[sfayer]

Hi @arrabito,

I believe this was fixed in v9.1.15: #8698 although it won't retroactively fix transformations/productions that were added on a broken version (v9.1.13 or v9.1.14). Is it possible that's the case here?

Regards,
Simon