Lua m.getvar()/m.getvars(): long transformation tables crash, and "none" doesn't reset in v3 #3737

@airween, here's another one from the #3730/#3731 code review. It's in the transformation argument of Lua m.getvar()/m.getvars(), i.e. Lua::applyTransformations() in v3 and resolve_tfns() in v2.

1. A long transformation table crashes the process (v3, confirmed)

The table loop calls lua_rawgeti(L, idx, i) once per entry and never pops the value (src/engine/lua.cc:425 on v3/master 9d45c6d). A C function only has LUA_MINSTACK (20) free stack slots unless it calls lua_checkstack(), so a long enough table writes past the Lua stack.

function main()
    local t = {};
    for i = 1, 100 do t[i] = "trim"; end
    local v = m.getvar("ARGS:a", t);
    m.log(4, "after: " .. tostring(v));
    return nil;
end

With SecRuleScript in phase 1 (macOS, Lua 5.5.1), regression_tests exits with a segmentation fault (exit 139) at 100 entries and above. 25 and 50 entries still work. The size only comes from the script, not from the request, but a script that builds the list dynamically, or just has a long one, takes the worker down.

v2's resolve_tfns() (apache2/msc_lua.c, around line 177) has the same loop without a lua_pop(). I haven't reproduced a crash there.

Fix: lua_pop(L, 1) after reading each name (on the continue path too), or copy the name and pop before using it.

2. "none" doesn't start over in v3 (behaviour differs from v2)

Both engines have the comment /* A "none" means start over */. v2 does that: it empties the list of transformations collected so far (tfn_arr->nelts = 0). v3 applies each transformation to the value as soon as it reads it, so its continue can't undo the ones already applied:

local v = m.getvar("ARGS:a", {"lowercase", "none"});   -- request ?a=ABC

v3 returns abc. Following v2 and the comment, it should be ABC. The fix would be to collect the transformation names first, resetting on "none" as v2 does, and only then apply them to a copy of the value.

3. luaL_checkstring() on a table entry can skip C++ destructors (v3)

If an entry isn't a string (e.g. {"lowercase", {}}), luaL_checkstring(L, -1) raises a Lua error with longjmp while getvar()'s std::string var and the loop's temporaries are alive. Their destructors don't run, so the value buffer leaks on every such call (and it's undefined behaviour if Lua is built as C++). This is the same pattern #3732 fixes in setvar(). Collecting the names first, as in point 2, before any std::string exists, would fix it too.

Points 1 and 2 need a regression case each. The 100-entry script above and the {"lowercase", "none"} one would do.