Right now, the extract_example feature of AutoDoc (which is in turn based on GAPDoc's features) is used by a bunch of packages to extract examples into .tst files which then are run in their CI tests. But it has some issues. One is an ugly choice: store the generated .tst files in the repo or not?
Usually the answer is clear: of course a generated file is not to be stored in the repo! But then it is very easy for a maintainer to run tests locally but forget to (re)generate those files. Also, in CI, one then has to make sure to run makedoc.g before running tests, something we currently don't want.
So a result, many packages now store those generated .tst files in the repo. It is not super bad, but it can also be a bit annoying: e.g. sometimes a change in AutoDoc can result in the XML it generates to slightly change; and then one gets diffs in the .tst files when re-running makedoc.g on a different computer with a different AutoDoc version.
The whole "generate .tst files" approach also makes it harder to fix failing tests, as from the tst file you are pointed at a generated XML file, which then points you at the actual place the test comes from, and where it has to be fixed. Automatic fixing of trivial test issues (e.g. changed adjusting to changed printing) also get more annoying than necessary (see issue #375).
An obvious alternative would be to extract the tests only at the start of tst/testall.g files; or not to extract them at all, but uses GAPDoc's RunExamples (which would also help a bit with issue #375).
But that's not quite as easy as it might sound: one of the things AutoDoc() does is to collect a list of all source files for the manual (including *.autodoc, *.xml, *.gd, *.gi, *.g files). For some projects this is trivial, but some have a more elaborate setup. We'd have to carefully replicate that; but we definitely do not want to specify this kind of information in two places (DRY principle). Also: GAPDoc does not know about AutoDoc comments and *.autodoc files, so we must involve AutoDoc somehow.
One solution that comes to mind is to introduce some kind of environment variable AUTODOC_EXTRACT_ONLY or so, which if set, causes AutoDoc() to not run the full machinery, but focus on extracting examples. Then one could perhaps run the equivalent of AUTODOC_EXTRACT_ONLY=1 gap makedoc.g at the start of tst/testall.g.
However, that still leaves another issue on the table I did not mention so far: running tests from a read-only package dir! Depending on how a user installed GAP (e.g. via a package manager like Debian's apt; or maybe a sysadmin installed it centrally), the package dirs may be read-only. And makedoc.g wants to write XML files it generated, and of course also the tst files with examples... That's a clash and it has bitten the utils package in the past, when it tried to call makedoc.g from its tst/testall.g.
One workaround for that could be to copy relevant files to a temp dir, and run stuff there, but that opens its own can of worms...
The best idea I had so far is to come up with a much enhanced version of ExtractExamples (and RunExamples), say AUTODOC_ExtractExamples, which combines what GAPDoc does with dedicated code for "extracting" examples from .autodoc files and AutoDoc comments, without producing/needing the intermediate XML files being generated (those intermediate XML files are a blocker for issue #375 right now as well). Alas, that's a major effort.
(This issue is strongly related to issue #375 and it may be best to fix them together, or at least with the other one in mind.)
Right now, the
extract_examplefeature ofAutoDoc(which is in turn based on GAPDoc's features) is used by a bunch of packages to extract examples into .tst files which then are run in their CI tests. But it has some issues. One is an ugly choice: store the generated .tst files in the repo or not?Usually the answer is clear: of course a generated file is not to be stored in the repo! But then it is very easy for a maintainer to run tests locally but forget to (re)generate those files. Also, in CI, one then has to make sure to run
makedoc.gbefore running tests, something we currently don't want.So a result, many packages now store those generated .tst files in the repo. It is not super bad, but it can also be a bit annoying: e.g. sometimes a change in AutoDoc can result in the XML it generates to slightly change; and then one gets diffs in the .tst files when re-running
makedoc.gon a different computer with a different AutoDoc version.The whole "generate .tst files" approach also makes it harder to fix failing tests, as from the tst file you are pointed at a generated XML file, which then points you at the actual place the test comes from, and where it has to be fixed. Automatic fixing of trivial test issues (e.g. changed adjusting to changed printing) also get more annoying than necessary (see issue #375).
An obvious alternative would be to extract the tests only at the start of
tst/testall.gfiles; or not to extract them at all, but uses GAPDoc'sRunExamples(which would also help a bit with issue #375).But that's not quite as easy as it might sound: one of the things
AutoDoc()does is to collect a list of all source files for the manual (including *.autodoc, *.xml, *.gd, *.gi, *.g files). For some projects this is trivial, but some have a more elaborate setup. We'd have to carefully replicate that; but we definitely do not want to specify this kind of information in two places (DRY principle). Also: GAPDoc does not know about AutoDoc comments and *.autodoc files, so we must involve AutoDoc somehow.One solution that comes to mind is to introduce some kind of environment variable
AUTODOC_EXTRACT_ONLYor so, which if set, causesAutoDoc()to not run the full machinery, but focus on extracting examples. Then one could perhaps run the equivalent ofAUTODOC_EXTRACT_ONLY=1 gap makedoc.gat the start oftst/testall.g.However, that still leaves another issue on the table I did not mention so far: running tests from a read-only package dir! Depending on how a user installed GAP (e.g. via a package manager like Debian's
apt; or maybe a sysadmin installed it centrally), the package dirs may be read-only. Andmakedoc.gwants to write XML files it generated, and of course also the tst files with examples... That's a clash and it has bitten theutilspackage in the past, when it tried to callmakedoc.gfrom itstst/testall.g.One workaround for that could be to copy relevant files to a temp dir, and run stuff there, but that opens its own can of worms...
The best idea I had so far is to come up with a much enhanced version of
ExtractExamples(andRunExamples), sayAUTODOC_ExtractExamples, which combines what GAPDoc does with dedicated code for "extracting" examples from .autodoc files and AutoDoc comments, without producing/needing the intermediate XML files being generated (those intermediate XML files are a blocker for issue #375 right now as well). Alas, that's a major effort.(This issue is strongly related to issue #375 and it may be best to fix them together, or at least with the other one in mind.)