Can we deserialize automagically with simdjson::ondemand

[lemire]

The simdjson::ondemand API is begging for reflection-based deserialization.

We have this ugly approach:

https://github.com/simdjson/simdjson/blob/master/doc/basics.md#adding-support-for-custom-types

It works, but it requires too much work from our users. It should be automagical. :-)

[FranciscoThiesen]

Will start investigating this a bit. Will keep updates posted here (@lemire)

[FranciscoThiesen]

Will open a branch on https://github.com/simdjson/simdjson for it

[FranciscoThiesen]

Let's keep this open and when we have the official C++26 spec and compiler support we can add it to simdjson, using the version check as @lemire recently used for leveraging concepts for having serialization of vector

[lemire]

The extractor syntax should help....

simdjson/simdjson#2247

It will be interesting to find out how it interacts with C++ reflection.

@the-moisrex

[the-moisrex]

The extractor syntax should help....

simdjson/simdjson#2247

It will be interesting to find out how it interacts with C++ reflection.

@lemire

Extract should work without modification.

In C++26, we could simply, instead of blowing up in the user's face with static_assert(false, ...) in ::get(T& out), we can check the type's each child and if we have tag_invokes for all of them, then we can set each one.

So, we simply need a tag_invoke of the most general case possible but restrict it so that its non static members MUST be convertible as well.

OR or AND:

We could use a little helper function with extract to relax even that "MUST" above and just simply ignore the non-static member fields of the type; for example:

struct users {
  int id;
  std::string username;
  std::ifstream profile_image; // no can do kinda field; gets ignored
} user;

struct admins {
   int id;
   std::string username;
} admin;
obj.extract(
  to{"user", ignore_unknown_fields(user)},
  to{"admin", admin} // no need to do anything, the global `tag_invoke` should be able to handle this
);

---

So, I don't think we need to do much about the extract idea, but we certainly can provide a general tag_invoke, or change the implementation of document/value::get member functions in order to let any type that all of its fields are convertible, be convertible.

Note: extract already ignores setting fields that JSON string doesn't provide, by ignore_unknown_fields I mean ignore the users' fields that we even can't convert to, even if the JSON string provides values for.

[the-moisrex]

@lemire, @FranciscoThiesen Here's a mock code of what I'm thinking:

namespace simdjson {

template <typename T>
consteval auto is_convertible_type() {
  for (std::meta::info field : nonstatic_data_members_of(^T)) {
    if (deserializable(type_of(field))) return false;
  }
  return true;
}

template <typename T>
  requires (is_convertible_type<T>()) // all of its fields MUST be deserializable as well
constexpr error_code tag_invoke(deserialize_tag, auto& value, T& out) {
  template for (constexpr auto e : std::meta::nonstatic_data_members_of(^T)) {
    simdjson::ondemand::object obj;
    if (auto error = value.get_object().get(obj); error) {
      return error;
    }
    // it might be better if we could somehow call .extract once
    obj.extract(
        to{name_of(e), [:e:]}
    );
  }
  return SUCCESS;
}

}

It's my first time writing C++26 reflections, so, I don't know how reflections work in above mock code.

Honestly, this could be laughably easy to implement if C++26's reflection is what I think it is.

[Yaraslaut]

I looked at simdjson PR and it looks amazing, the issue with disserealization and reflection is the need to call recursion on objects that are not trivial, so you need to call sometimes to and sometimes sub in terms of simdjson PR

Here is idea that i used to make simple deserializer for yaml-cpp where it provides map-like structure via Node

template <typename T> void from_node(auto const &node, T &t) {

  util::for_range<0, util::number_of_members<T>()>([&]<auto I>() { // loop over members
    constexpr auto mem = util::member_info<T>(I);
    auto name = std::string{util::name_of(mem)};
    if constexpr (std::is_constructible_v<std::formatter<[:type_of(mem):]>>) { // check if type is trivial like double, int, char ...
      t.[:mem:] = node[name].template as<[:type_of(mem):]>());
    } else {
      from_node(node[name], t.[:mem:]); // call recursion when member is some other structure
    }
  });

Also, when you use reflection to deserialize it is not clear how to handle custom cases like this one

      to{"year", [&car](auto val) {
        car.year = val;
      }});