fix: macro kwargs, unary operators, min/max attribute, indent, format filter and llama.cpp 7fe450e1 parity - #20
Merged
Conversation
…a.cpp parity gaps
…arity; regenerate cross test
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14, closes #15, closes #16, closes #17, closes #18, closes #21, closes #22.
Causes and fixes
Not enough arguments provided to 'f'(llama.cpp's rule and message). The same binding now applies tocaller(...)parameters of a{% call %}block (keywords, defaults and the missing-argument error were ignored before).-/+. The lexer already emitted unary tokens, but the parser had no rule for them. Added llama.cpp'sparse_unary_expressionbetween filters and call/member, so-n|absis(-n)|abs,-n is numberis(-n) is number,-a ** 2is(-a) ** 2(Jinja2's precedence). The lexer also treats-/+after thenotoperator as unary (not after.notor|not).min/max(attribute=). Returned the attribute value; now return the item, and readattributepositionally too.a[]parses to a newBlankExpression(exported fromast.dart) that is undefined;int * strrepeats;indentaccepts a string width;str.formatsupports{}fields with llama.cpp's input marking. Other fields ({0},{name},{{,{:>5}) throwformat() only supports simple '{}' placeholders, as llama.cpp does.skip:. Adjacent C++ string literals are now joined correctly; before, 4 multi-line templates kept" "junk." "from template text, a workaround for the generator bug above. Removed. Nothing else depended on it: the only other occurrences (two Hermes fixtures) are inside{{ }}string literals, which the workaround never touched.indent. Rewritten on llama.cpp's line model: a trailing newline is kept once (plus the indent withblank=true), so'foo\n'|indentisfoo\n. Two exceptions. llama.cpp drops leading empty lines whenfirst=false('\nfoo'|indentisfoo, an obvious bug), so those follow Jinja2 (\n foo), as dinja already did.''|indent(2, true)is''as in llama.cpp (Jinja2 gives' ').a is divisibleby -aalready parses as(a is divisibleby) - a, as it does in llama.cpp and Jinja2. Neither engine accepts-as the start of an unparenthesized test argument. Both then throw, becausedivisiblebygot no argument; dinja returnedFalseand rendered-2. The issue's expectedTrueis not what either engine gives. Tests that take an argument (divisibleby,eq/equalto,ne,gt/greaterthan,ge,lt/lessthan,in, plus dinja'sle,sameas,startingwith,endingwith,ieq) now throwTest expected 2 arguments, got 1without one, including throughselect('eq').*unpacking. llama.cpp parsesf(*x)but has no runtime for it (cannot exec SpreadExpression). Following that, dinja now throwsArgument unpacking with * is not supportedinstead of passing the list as one argument. Jinja2 would unpack.formatfilter. In llama.cpp,'...'|format(...)reaches the same{}-only stringformatbuiltin, so dinja's new filter callsstr.format:'{}-{}'|format(1, 2)is1-2and'%s-%s'|format(1, 2)is%s-%s, as in llama.cpp. Non-strings throwUnknown filter 'format' for type Integer, as llama.cpp does. Jinja2's%-formatting ('%s-%s'|format(1, 2)gives1-2) is not implemented.example/security_example.dartnow wrap user input inJinjaString.user, and say that plain strings are not escaped. Example output:Decisions where llama.cpp and Jinja2 differ
f()forf(a, b=2)Not enough arguments[|2]f(1, z=3)[1|](count bug)takes no keyword argument 'z'f(1, a=2)[2|]got multiple values for argument 'a'f(b=5)[|2](dropsb=5)[|5][|5]f(1, 2, 3)[1|2][1|2](unchanged)not -n,-a ** 2,-b // a**///)False,4,-3n.1on an int''''(same asn[1])'<{}>'.format(undefined)<><>'a}b'.format()a}ba}b'\nfoo'|indentfoo(drops the line)\n foo''|indent(2, true)''' 'f(*[1, 2, 3])cannot exec SpreadExpression1|2|3'%s-%s'|format(1, 2)%s-%s1-2The #14 brief asked for errors on unknown and duplicate keywords "the way llama.cpp does". llama.cpp doesn't raise those errors; it binds them wrong. This PR throws as Jinja2 does.
varargs/kwargsare still unsupported, as in llama.cpp. A macro whose body useskwargsand is called with extra keywords now throws; before, it renderedkwargsas empty. No fixture does this.min/maxstay case-sensitive, as in llama.cpp. dinja has nocase_sensitivesupport, so Jinja2's case-insensitive default isn't matched.Evidence
llama.cpp outputs come from its
common/jinjabuilt standalone at 7fe450e1. Jinja2 3.1.6 outputs come from aSandboxedEnvironmentwith the same options astest-jinja.cpp -py. Each of the 175 expectations in the new tests matches the engine its comment names.Tests
179 new cases. 146 fail on
main(3ff742e) and pass here. 33 are labelled guards ("unchanged", "Binary minus is unchanged", "still throws", "stays binary") and pass on both. The guards cover binary minus,b - -k,b - -2, negative literals,(k) -b,lst[1] -k,k -}},d.not - 1andd|not - 1, all with unchanged trees. Two existing tests that assertedmin/max(attribute=)returning the value now assert the item.Mutations (all killed unless noted)
-(n|abs))-m|abs, tree testa[]not blank-afternotbinary-after.not/|notunaryint * strremovedint * strdrops input markingcallercounted as an argumentcallerrejected as unknown keywordcaller(...)parameters unboundmin/maxreturn the valueattributeignoredmaxtie keeps the last itemformatremovedformatdrops literal / argument markingformataccepts other fieldsBlankExpressionevaluates to noneBlankExpressionevaluates to0" "againindentkeeps the trailing empty line*passes the value throughformatfilter removedformatfilter stringifies non-stringsCross-test regeneration
test/llama_cross_test.dartis regenerated from 7fe450e1'stests/test-jinja.cpp. The old file had 243 cases; the new one has 290. The generated names, templates, data, expectations and skip flags match, for all 290, the cases dumped from upstream's compiledtest-jinja(279 pass, 11 skip, 0 fail).mainwith the same file: 262/279 pass. The 17 failures are the Macro keyword arguments in a different order render empty values #14, Unary minus/plus on a variable fails to parse (including slice bounds) #15 and Remaining parity gaps with llama.cpp 7fe450e1's Jinja tests #17 cases.--run-skipped, dinja passes 8 of the 11 not-implemented cases. The 3formatforms throw by design.Consumer check (llamadart c0dbdc1d6, throwaway copy,
dependency_overridesonly)Current llamadart
origin/mainis 269d337f4. It differs from c0dbdc1d6 inlib,testandpubspec.yamlonly by doc comments and a CI script path.test/unit/core/templateandtest/integration/core/template: identical with dinjamainand with this branch: 757 pass, 74 skip, 2 fail. The 2 failures (tool_call_fixture_render_test: Llama 3.2, Ministral 3) also fail onmain. They come from 1.1.1'stojsonspacing; llamadart pins<1.1.0.ChatTemplateEnginescenarios each plusJinjaAnalyzercaps: 464 results, 0 differences. A static scan of 185 macro calls in those templates finds none that the new binding errors reject.Gates
dart formatclean,dart analyzeno issues,dart test892 pass / 11 skip, coverage 90.17%,dart test -p chrome test/src test/llama_cross_test.dart790 pass / 11 skip,dart pub publish --dry-run0 warnings, pana 160/160.Unrelated, not fixed (llama.cpp differs, dinja matches Jinja2)
{{ [1, 2]|select('eq', 2)|list }}: llama.cpp prints2, dinja and Jinja2 print[2].{{ '{}'.format(x) }}withxundefined: llama.cpp throwsUndefined (hint: 'x') is not a string value, dinja and Jinja2 print nothing.{{ n.1 }}on an integer: llama.cpp throwsCannot access property with non-string: got Integer, dinja and Jinja2 print nothing.