Support Monitor Layout types + PostgreSQL Example by lukelowry · Pull Request #616 · ORNL/GridKit

lukelowry · 2026-10-11T13:18:02Z

Description

Adds an output_layout solver option for monitor output. With "output_layout": "long", each row of output_file holds one variable at one sample instead of one column per variable, so results load straight into SQL tables no matter how many variables are monitored.

Closes #597.

Wide output is unchanged, and printing is faster:

Format	`develop`	This PR
CSV	0.199 s	0.156 s
JSON	0.377 s	0.179 s
YAML	0.310 s	0.181 s

Print time only, 2000 monitors × 4 variables over 500 samples.

Proposed changes

Add VariableMonitorLayout (WIDE, LONG) to SinkSpec, set by output_layout next to output_file in the solver file. It defaults to wide, and long is not supported with reference_file.
Long rows are sample, t, class, id, variable, value rather than the t, id, value sketched in the issue:
- class and id are the device's class and id from the case file, since an id alone can repeat across classes.
- variable gets its own column, so queries can filter or group by variable without parsing names.
- sample is needed for a unique key, since the samples just before and after an event share the same t.
Monitors hand their variables to the controller once (appendEntries), and the controller writes every format and layout from that list. This replaces the per-format virtuals in VariableMonitorBase and the formatting code in VariableMonitor.
The layout is a template parameter of each sink (Sink<FormatT, Layout>), resolved at compile time.
Monitored getters must return a real value (static_assert in set()); every existing getter does.
Remove the unused VariableMonitorController::addVariable.
Add a DynamicSimulation/PostgreSQL example that runs an ACTIVSg2000 bus fault and loads the long output into PostgreSQL with Docker Compose or Podman.
Mark case-file monitors for removal in a follow-up PR.

Checklist

All tests pass.
Code compiles cleanly with flags -Wall -Wpedantic -Wconversion -Wextra.
The new code follows GridKit™ style guidelines.
There are unit tests for the new code.
The new code is documented.
The feature branch is rebased with respect to the target branch.
The CHANGELOG.md has been updated to reflect the changes.

Further comments

@pelesh Feedback on PR length/scope

@tsybinae Thank you for the suggestion and guidance.

tsybinae · 2026-10-15T19:27:06Z

Hi. I am reviewing the code. It builds, runs, and produces the csv. The timestamp looks a bit odd, for instance, row 569215 looks like this

1704,7.0916666666666668e+00,Genrou,5281_1_genrou,omega,2.2408284627934353e-05

This effectively means that t = 7.09 something. Is this thing going to merge on t if I need to left join?

Thank you.

lukelowry · 2026-10-16T02:17:45Z

@tsybinae Thanks! Join on sample, since t repeats at events (before and after). Also, the time just has a rounding issue due to floating point precision, so joining on t is not recommended.

Let me know if you have any othr issues!

tsybinae · 2026-10-16T12:58:18Z

This functionality is not specific to the csv task, so the merge can go ahead. There may be more precision related questions within a few month.

tsybinae

Builds and runs cleanly.

superwhiskers

core idea seems like the right approach. just some comments on implementation of formatting that should be addressed before this goes further.

superwhiskers · 2026-10-16T15:06:27Z

-      void append(std::string& out, Json json) const override
+      void append(std::string& out, Json json) const
      {
-        std::ostringstream os;
-        os.precision(std::numeric_limits<RealT>::digits10 + 1);
-        os << std::scientific;
-
-        std::string indent = "  ";
        if (json.after_first)
        {
-          os << indent << ",\n";
+          out += "  ,\n";
        }
-        os << indent << "{\n";
-        indent.append(2, ' ');
+        out += "  {\n    \"t\": ";
+        VariableMonitorDetail::formatReal(out, *time_);

-        os << indent << std::quoted("t") << ": " << *time_ << ",\n";
-        for (auto&& var : variables_)
+        const Entry* previous = nullptr;
+        for (auto&& entry : entries_)
        {
-          os << indent << std::quoted(var.label) << ": " << *var.value << ",\n";
+          if (previous && previous->device_class == entry.device_class && previous->id == entry.id)
+          {
+            out += ",\n      \"";
+          }
+          else
+          {
+            if (previous)
+            {
+              out += "\n    }";
+            }
+            out += ",\n    \"";
+            out += entry.device_class;
+            out += '_';
+            out += entry.id;
+            out += "\": {\n      \"";
+          }
+          out += entry.variable;
+          out += "\": ";
+          VariableMonitorDetail::formatReal(out, entry.value());
+
+          previous = &entry;
+        }
+        if (previous)
+        {
+          out += "\n    }";
        }
+        out += "\n  }";
+      }


since we are already pulling in nlohmann/json, i would recommend using its json serialization capabilities here. that would make this easier to read

I don't think it is a good idea to pull third-party dependency in the GridKit code beyond the JSON parser. Variable monitoring and JSON formatting should be cleanly separated so that we can replace parser dependencies (if needed) with other library.

i disagree. with how we currently write the data, when using the json output format for monitored variables, if a component has its id set to a string which contains an escaped double quote, it will be unescaped in the output, creating malformed json output. if we used an existing json writer, this problem is solved. presumably, this would also apply to csv output and unescaped commas, or something similar for yaml

example modified case file:

example malformed json output:

the same also applies to newlines and tabs. i.e. \n\t in the id of a component will output a literal newline and tab, so it's not just a matter of escaping a " in the id of a component. you need to handle a lot more

superwhiskers · 2026-10-16T15:08:02Z

+        out += "  - t: ";
+        VariableMonitorDetail::formatReal(out, *time_);
+        out += '\n';

-        auto after_first = false;
-        for (auto* mon : monitors_)
+        const Entry* previous = nullptr;
+        for (auto&& entry : entries_)
        {
-          if (after_first)
+          if (!previous || previous->device_class != entry.device_class || previous->id != entry.id)
          {
-            os << ",\n";
+            out += "    ";
+            out += entry.device_class;
+            out += '_';
+            out += entry.id;
+            out += ":\n";
          }
-          std::string monitor_out;
-          mon->append(monitor_out, Json());
-          os << monitor_out;
-          after_first = true;
-        }
-
-        indent.erase(indent.size() - 2);
-        os << '\n'
-           << indent << "}";
+          out += "      ";
+          out += entry.variable;
+          out += ": ";
+          VariableMonitorDetail::formatReal(out, entry.value());
+          out += '\n';

-        out += os.str();
+          previous = &entry;
+        }


we don't have a yaml library so i understand doing this manually, but it could be improved by using <format> to avoid repeatedly concatenating new strings to the output

Each += appends into one reused buffer, so no new strings are created after the first sample. I tried std::format_to for YAML, but 1.9× slower for wide and 2.2× slower for long layout

nkoukpaizan · 2026-10-16T15:39:12Z

-       */
-      virtual void appendHeader(std::string&, Csv) const = 0;
+      /// Returns the current value of a monitored variable
+      using GetterT = std::function<double()>;


Everything is now converted to double before printing, including float, int, bool, etc.

Attempted to resolve, but I might be misunderstanding. Please advise

lukelowry · 2026-10-18T13:27:35Z

@nkoukpaizan @superwhiskers feedback greatly appreciated! the example I hope is much better. And hopfully the template parameter approach is cleaner, please LMK

lukelowry requested review from pelesh, superwhiskers and tsybinae October 11, 2026 13:18

lukelowry self-assigned this Oct 11, 2026

lukelowry added the enhancement New feature or request label Oct 11, 2026

lukelowry force-pushed the lukel/monitor-long-dev branch from 54bd651 to bba5a78 Compare October 13, 2026 22:04

Support Layout types WIDE and LONG

757dcfe

lukelowry force-pushed the lukel/monitor-long-dev branch from bba5a78 to 757dcfe Compare October 14, 2026 22:24

improve output

bed6f96

tsybinae approved these changes Oct 16, 2026

View reviewed changes

superwhiskers requested changes Oct 16, 2026

View reviewed changes

nkoukpaizan reviewed Oct 16, 2026

View reviewed changes

lukelowry added 4 commits October 18, 2026 05:20

move layout setting to the .solver.json and make layout a tempalte param

a2ba9e5

docker ocmpose and rename

7295526

update example README

197b6af

replace switch after template change

f81fbde

lukelowry requested review from nkoukpaizan and superwhiskers October 18, 2026 13:26

Conversation

lukelowry commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Proposed changes

Checklist

Further comments

Uh oh!

tsybinae commented Oct 15, 2026

Uh oh!

lukelowry commented Oct 16, 2026

Uh oh!

tsybinae commented Oct 16, 2026

Uh oh!

tsybinae left a comment

Choose a reason for hiding this comment

Uh oh!

superwhiskers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukelowry commented Oct 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lukelowry commented Oct 11, 2026 •

edited

Loading