fix: keep input marking through filters, tojson and join; undefined member access

CHANGELOG.md

@@ -1,3 +1,13 @@
+## Unreleased
+
+- Fixed a `{% filter %}` block, and a filter on the output of a block `set`, a macro or `caller()`, getting `JinjaString.user` text already escaped: with `x = JinjaString.user('<b>')`, `{% filter upper %}{{ x }}{% endfilter %}` is now `&lt;B&gt;`, not `&LT;B&GT;`, and `{% filter length %}{{ x }}{% endfilter %}` is `3`, not `9`, as in llama.cpp. Input text is now escaped once, when the template outputs it. `| safe` on such output, `{% filter safe %}`, and `| safe` on text made from it or from other safe text escape the input in it, as Jinja2 with autoescape does; `| safe` on a value passed to `render`, or returned by a function passed to it, still outputs it unescaped. A block whose output also contains `| safe` input text is still escaped before the filter runs, as `~` does.
+- Fixed `tojson` and `join` dropping the input marking, so `JinjaString.user` text in their output was not escaped: `{{ {"k": x} | tojson }}` is now `{"k": "&lt;b&gt;"}` and `{{ [x, "<i>"] | join }}` is `&lt;b&gt;<i>`. Only input text is escaped, after JSON escaping, in values, dict keys and separators; the JSON's own quotes and template text are not. `tojson` output with input text in it is no longer safe (`is escaped`).
+- Fixed input text being output unescaped by string indexing, `first`, `last`, `reverse`, `list` and `for` on a string, `split` and `rsplit` pieces, `strip`, `lstrip` and `rstrip` with characters, `truncate`, `str()`, `~`, `+` and `format` with a list or dict, `string`, `upper`, `lower`, `title`, `capitalize`, `trim`, `replace` and `indent` on a list or dict, `replace`, `indent` and `format` on text that is only partly input, input passed as a `replace` replacement, input dict keys in a `for` loop, and an input `strftime_now` format.
+- Changed `replace` and `indent` on input text to leave the template text they add unescaped: `x | replace('b', '<i>')` is now `&lt;<i>&gt;`, not `&lt;&lt;i&gt;&gt;`, and `x | indent('> ', true)` no longer escapes `> `. Each character keeps the marking of the text it came from; llama.cpp marks all of the result as input when all of the source is.
+- Fixed `| safe` text being escaped when passed to `format`, or to `replace` or `indent` on input text, sliced with a step or split by `unique`, and escaped twice when it held escaped input: `'{}'.format(x ~ (y | safe))` is now `&lt;b&gt;<i>`, not `&amp;lt;b&amp;gt;<i>`.
+- Fixed member access on an undefined value giving a function, which printed as `<function NAME>` and was `defined`: for an undefined `y`, `{{ y.text }}` and `{{ y[0]['text'] }}` now print nothing and are not `defined`, as in llama.cpp. As in llama.cpp, `y.upper`, `y.length` and the other filters that llama.cpp gives undefined are functions returning an empty value (`y.length()` is `0`, `y.default(1)` is `1`) that print nothing, and calling any other member, such as `y.get()`, throws. Devstral Small 2507 no longer prints `<function text>` for an assistant turn without `content`. Also as in llama.cpp, Command R+ tool use renders an `object` parameter without `additionalProperties` as `Dict[str, Union[]]`, not `Dict[str, Any]`, and QwQ-32B throws for an assistant tool-call turn without `content` before the last turn.
+- Changed a number subscript of none, as in `x[0]` or `x[0]['text']` for `x = none`, to throw `Cannot access property with non-string: got Integer`, as llama.cpp does; it gave undefined. Devstral Small 2507 with an assistant turn whose content is none now throws, as in llama.cpp.
+
 ## 1.2.0
 
 - Rewrote the README and example around rendering a chat template, and updated the pubspec description and topics.

README.md

@@ -93,6 +93,8 @@ Real templates come from the `chat_template` field of a model's `tokenizer_confi
 
 Wrap untrusted values in `JinjaString.user`. They are escaped when rendered; a plain `String` is treated as template text and is not.
 
+Input text keeps its marking through filters, `~`, loops, macros and `{% set %}` and `{% filter %}` blocks, and is escaped once, when the template outputs it. So a filter sees it as it was passed in: `{% filter length %}{{ name }}{% endfilter %}` counts the characters of `name`, not of its escaped form. `tojson` and `join` escape only the input text in their output, not the JSON's quotes or the template's separator. `| safe` and `markSafe()` turn escaping off for a value passed to `render`, or returned by a function passed to it. The output of a block `set`, macro, `caller()` or `{% filter %}` block is final, as in Jinja2 with autoescape, so `| safe` on it escapes the input in it.
+
 ```dart
 import 'package:dinja/dinja.dart';
 

lib/src/ast/nodes.dart

@@ -1,5 +1,6 @@
 import '../types/value.dart';
 import '../types/jinja_string.dart';
+import '../types/marking.dart';
 import '../types/repr.dart';
 import 'dart:math' as math;
 import '../runtime/context.dart';
@@ -9,7 +10,7 @@ import '../lexer.dart'; // for Token
 /// Thrown by `break` and `continue`. Carries the output that the loop pass
 /// rendered before the statement, which the loop keeps.
 abstract class LoopSignal {
-  final List<JinjaStringPart> parts = [];
+  final List<JinjaString> output = [];
 }
 
 class BreakSignal extends LoopSignal {}
@@ -48,63 +49,68 @@ class Program extends Statement {
   @override
   String get type => 'Program';
 
+  /// Renders the template. Input-marked text is escaped here, once.
   @override
   JinjaValue execute(Context ctx) {
-    return execStatements(body, ctx);
+    final result = renderScope(
+      ctx.environment.values,
+      () => execStatements(body, ctx),
+    );
+    return JinjaStringValue((result as JinjaStringValue).value.escape());
   }
 }
 
-// ... helper to execute list of statements and gather output ...
-/// Executes a list of [stmts] within the given [ctx] and aggregates their results.
+/// Executes a list of [stmts] within the given [ctx] and joins their output.
 ///
 /// If a statement returns a string value, it is added to the output.
 /// Non-string values (except None and Undefined) are converted to strings.
+/// Input-marked text stays unescaped, so a filter applied to the output sees
+/// it as it was passed in; see [_joinOutput].
 JinjaValue execStatements(List<Statement> stmts, Context ctx) {
-  List<JinjaStringPart> parts = [];
+  final output = <JinjaString>[];
   try {
     for (final stmt in stmts) {
       final val = stmt.execute(ctx);
       if (val is JinjaStringValue) {
-        if (val.isSafe) {
-          parts.addAll(val.value.parts);
-        } else {
-          // Check parts for input/unsafe content
-          for (final part in val.value.parts) {
-            if (part.isInput) {
-              final escaped = part.val
-                  .replaceAll('&', '&amp;')
-                  .replaceAll('<', '&lt;')
-                  .replaceAll('>', '&gt;')
-                  .replaceAll('"', '&quot;')
-                  .replaceAll("'", '&#39;');
-              // print('DEBUG escaping: ${part.val} -> $escaped');
-              parts.add(
-                JinjaStringPart(escaped, part.isInput),
-              ); // Escaped -> safe/template
-            } else {
-              parts.add(part);
-            }
-          }
-        }
+        output.add(val.value);
       } else if (!val.isNone && !val.isUndefined) {
-        parts.addAll(reprOf(val).escape().parts);
+        output.add(reprOf(val));
       }
     }
   } on LoopSignal catch (signal) {
-    signal.parts.insertAll(0, parts);
+    signal.output.insertAll(0, output);
     rethrow;
   }
-  return JinjaStringValue(JinjaString(parts, isSafe: true));
+  return JinjaStringValue(_joinOutput(output));
+}
+
+/// Joins rendered [pieces].
+///
+/// Input-marked text stays unescaped unless a piece is a safe string with
+/// input-marked text, whose text is final: then every piece is escaped and
+/// the result is safe.
+JinjaString _joinOutput(List<JinjaString> pieces) {
+  final safeInput = pieces.any(
+    (p) => p.isSafe && p.parts.any((part) => part.isInput),
+  );
+  if (pieces.any(hasRawInput) && !safeInput) {
+    return JinjaString([for (final p in pieces) ...p.parts]);
+  }
+  return JinjaString([
+    for (final p in pieces) ...p.escape().parts,
+  ], isSafe: true);
 }
 
 /// Executes [stmts] whose output is captured rather than printed, as in a
 /// block `set`, a `filter` block, a macro or a `caller()` body. A `break` or
 /// `continue` leaving the body discards the captured output, as in llama.cpp.
 JinjaValue _execCaptured(List<Statement> stmts, Context ctx) {
   try {
-    return execStatements(stmts, ctx);
+    final result = execStatements(stmts, ctx) as JinjaStringValue;
+    markRendered(result.value);
+    return result;
   } on LoopSignal catch (signal) {
-    signal.parts.clear();
+    signal.output.clear();
     rethrow;
   }
 }
@@ -194,22 +200,23 @@ class ForStatement extends Statement {
       // Sort keys for deterministic output if needed, though Jinja2 doesn't strictly guarantee order, usually tests expect sorted keys or we should sort.
       // Standard Jinja2 iterates keys.
       // llama.cpp tests seem to expect keys.
-      final keys = iterableVal.asMap.keys.toList();
-      // keys.sort(); // Optional: sort keys? Dinja existing tests expect 'a,b'. Maps in Dart preserve insertion order? LinkedHashMap does.
-      // If the map came from JSON/Literal, it's LinkedHashMap.
-
-      items = keys.map((k) => JinjaStringValue.fromString(k)).toList();
+      // Keys with the same string form are iterated once, as in asMap.
+      final keys = <String, JinjaValue>{
+        for (final k in iterableVal.items.keys) k.toString(): k,
+      };
+      items = [
+        for (final MapEntry(:key, :value) in keys.entries)
+          value is JinjaStringValue && hasRawInput(value.value)
+              ? value
+              : JinjaStringValue.fromString(key),
+      ];
     } else if (iterableVal is JinjaList) {
       items = iterableVal.items;
     } else if (iterableVal is JinjaTuple) {
       items = iterableVal.items;
     } else if (iterableVal is JinjaStringValue) {
-      // Iterate chars
-      items = iterableVal.value.parts
-          .map((p) => p.val)
-          .expand((s) => s.split(''))
-          .map((c) => JinjaStringValue.fromString(c))
-          .toList();
+      items = charsOf(iterableVal.value);
+      deriveRendered(JinjaList(items), [iterableVal]);
     }
 
     // Filter items first if testExpr
@@ -228,7 +235,7 @@ class ForStatement extends Statement {
     }
 
     // Loop
-    List<JinjaStringPart> parts = [];
+    final output = <JinjaString>[];
     // As in Jinja2 and llama.cpp, the else block runs unless a pass finishes
     // without `break` or `continue`.
     var iterated = false;
@@ -257,28 +264,28 @@ class ForStatement extends Statement {
 
       try {
         final result = execStatements(body, loopCtx);
-        if (result is JinjaStringValue) parts.addAll(result.value.parts);
+        if (result is JinjaStringValue) output.add(result.value);
         iterated = true;
       } on ContinueSignal catch (signal) {
-        parts.addAll(signal.parts);
+        output.addAll(signal.output);
       } on BreakSignal catch (signal) {
-        parts.addAll(signal.parts);
+        output.addAll(signal.output);
         break;
       }
     }
 
     if (!iterated) {
       try {
         final result = execStatements(defaultBlock, ctx);
-        if (result is JinjaStringValue) parts.addAll(result.value.parts);
+        if (result is JinjaStringValue) output.add(result.value);
       } on LoopSignal catch (signal) {
         // A `break` or `continue` in the else block belongs to an enclosing
         // loop and keeps the output of this one, as in Jinja2.
-        signal.parts.insertAll(0, parts);
+        signal.output.insertAll(0, output);
         rethrow;
       }
     }
-    return JinjaStringValue(JinjaString(parts, isSafe: true));
+    return JinjaStringValue(_joinOutput(output));
   }
 
   void _bindLoopVar(Context ctx, Expression loopVar, JinjaValue item) {
@@ -714,9 +721,25 @@ class MemberExpression extends Expression {
   @override
   String get type => 'MemberExpression';
 
-  @override
   @override
   JinjaValue execute(Context ctx) {
+    JinjaValue? obj;
+    final result = _access(ctx, (value) => obj = value);
+    final source = obj;
+    if (source == null) return result;
+    if (result is JinjaFunction) {
+      if (isSupplied(result) || !RenderInputs([source]).holdsFinal) {
+        return result;
+      }
+      return JinjaFunction(result.name, (args, kwargs) {
+        final inputs = RenderInputs([source, ...args, ...kwargs.values]);
+        return inputs.derive(result.handler(args, kwargs));
+      });
+    }
+    return deriveRendered(result, [source]);
+  }
+
+  JinjaValue _access(Context ctx, void Function(JinjaValue) onObject) {
     // Handle slice expressions first, as they are a special case of computed access
     if (computed && property is SliceExpression) {
       final slice = property as SliceExpression;
@@ -725,6 +748,7 @@ class MemberExpression extends Expression {
       final stepVal = slice.step?.execute(ctx);
 
       final obj = object.execute(ctx);
+      onObject(obj);
 
       int? start = (startVal != null && !startVal.isNone)
           ? startVal.asInt
@@ -789,14 +813,15 @@ class MemberExpression extends Expression {
           return JinjaStringValue(s.substring(start, stop));
         }
 
+        final raw = rawOf(s);
         final parts = <JinjaStringPart>[];
         if (step > 0) {
           for (int i = start; i < stop; i += step) {
-            if (i >= 0 && i < len) parts.addAll(s[i].parts);
+            if (i >= 0 && i < len) parts.addAll(raw[i].parts);
           }
         } else if (step < 0) {
           for (int i = start; i > stop; i += step) {
-            if (i >= 0 && i < len) parts.addAll(s[i].parts);
+            if (i >= 0 && i < len) parts.addAll(raw[i].parts);
           }
         }
         return JinjaStringValue(JinjaString(parts));
@@ -808,10 +833,19 @@ class MemberExpression extends Expression {
     if (computed || property is IntegerLiteral) {
       // Bracket notation: obj[expr], or an index after a dot: obj.0
       final obj = object.execute(ctx);
+      onObject(obj);
       final prop = property.execute(ctx);
       if (!computed && prop.asInt < 0) {
         throw Exception('Static member property cannot be negative');
       }
+      // As in llama.cpp, a subscript of undefined is undefined, and none
+      // takes only a string subscript.
+      if (obj.isUndefined || prop.isUndefined) return const JinjaUndefined();
+      if (obj.isNone && prop is! JinjaStringValue) {
+        throw Exception(
+          'Cannot access property with non-string: got ${prop.typeName}',
+        );
+      }
 
       if (obj is JinjaMap) {
         // Bracket access: check keys FIRST
@@ -851,7 +885,7 @@ class MemberExpression extends Expression {
           String s = obj.value.toString();
           if (idx < 0) idx += s.length;
           if (idx >= 0 && idx < s.length) {
-            return JinjaStringValue.fromString(s[idx]);
+            return JinjaStringValue(rawOf(obj.value)[idx]);
           }
         }
       }
@@ -869,6 +903,7 @@ class MemberExpression extends Expression {
       }
       final propName = (property as Identifier).name;
       final obj = object.execute(ctx);
+      onObject(obj);
 
       // Priority: 1. Attributes/Methods, 2. Map keys, 3. Special cases (length)
       final method = resolveMember(obj, propName);
@@ -942,7 +977,10 @@ class CallExpression extends Expression {
       }
     }
 
-    return func.handler(positionals, kwargs);
+    return deriveRendered(called(func, func.handler(positionals, kwargs)), [
+      ...positionals,
+      ...kwargs.values,
+    ]);
   }
 }
 
@@ -1004,17 +1042,23 @@ class BinaryExpression extends Expression {
         }
         // String repeat? `~` is concat. `*` is repeat in Python.
         if (l is JinjaStringValue && r.isNumeric) {
-          return JinjaStringValue(
-            JinjaString([
-              for (var i = 0; i < r.asInt; i++) ...l.value.parts,
-            ], isSafe: l.isSafe),
+          return deriveRendered(
+            JinjaStringValue(
+              JinjaString([
+                for (var i = 0; i < r.asInt; i++) ...l.value.parts,
+              ], isSafe: l.isSafe),
+            ),
+            [l],
           );
         }
         if (l is JinjaInteger && r is JinjaStringValue) {
-          return JinjaStringValue(
-            JinjaString([
-              for (var i = 0; i < l.value; i++) ...r.value.parts,
-            ], isSafe: r.isSafe),
+          return deriveRendered(
+            JinjaStringValue(
+              JinjaString([
+                for (var i = 0; i < l.value; i++) ...r.value.parts,
+              ], isSafe: r.isSafe),
+            ),
+            [r],
           );
         }
         throw Exception('Invalid operand types for *');
@@ -1184,7 +1228,7 @@ class FilterExpression extends Expression {
 
     if (globalFilters.containsKey(filterName)) {
       final res = globalFilters[filterName]!(args, kwargs);
-      return res;
+      return deriveRendered(res, [...args, ...kwargs.values]);
     }
     throw Exception('Unknown filter: $filterName');
   }
@@ -1240,14 +1284,22 @@ class TestExpression extends Expression {
   }
 }
 
-JinjaStringValue _concat(JinjaValue l, JinjaValue r) {
-  var left = l is JinjaStringValue ? l.value : JinjaString.template('$l');
-  var right = r is JinjaStringValue ? r.value : JinjaString.template('$r');
-  if (left.isSafe != right.isSafe) {
-    left = left.escape();
-    right = right.escape();
-  }
-  return JinjaStringValue(left + right);
+JinjaValue _concat(JinjaValue l, JinjaValue r) =>
+    deriveRendered(_join(l, r), [l, r]);
+
+JinjaStringValue _join(JinjaValue l, JinjaValue r) {
+  final left = _operand(l);
+  final right = _operand(r);
+  if (left.isSafe == right.isSafe) return JinjaStringValue(left + right);
+  final joined = _joinOutput([left, right]);
+  if (!joined.isSafe) return JinjaStringValue(joined);
+  return JinjaStringValue(left.escape() + right.escape());
+}
+
+JinjaString _operand(JinjaValue v) {
+  if (v is JinjaStringValue) return v.value;
+  final s = stringOf(v);
+  return hasRawInput(s) ? s : JinjaString.template('$v');
 }
 
 /// Represents a selection expression used in filtering iterations.