Repository navigation
Replies: 36 comments 70 replies
|
Yes, it outlines the same problems, and also suggests flatbuffers. My question is mostly, is there anything Im overlooking when using websockets as a transport? |
|
Moving halui into task and bypassing NML would be a good thing. The real problem currently in NML is the fact that error messages are not available on all readers and errors can therefore not be correlated correctly over all connections. This needs to be rethought, how errors are generated, propagated and correlated. I'm not to thrilled about using boost derived packages. It is very heavy and most of it, if not all, has been made redundant in modern C++. Actually, I'd rather have the boost dependency removed completely. Weren't you a fan of pybind11? ;-) With the proper interface, a "view" of HAL can be made visible remotely. As long as it stays within the confines of "slow as hell will do", then it should work. However, using a different interface structure would probably be better and leave HAL completely on the server-/machine-side. Using websockets, just like many other network communication layers, is how to handle errors. Errors are very much more recurring on network connections than any local interface. How to handle state-full information and sync/recovery is the art of the beast, so to say. |
|
talking of which, I have a branch that removes boost-python dependency, replacing it with pybind11. currently UIs are a hal userspace component, and that is what killed remote UIs. (it was possible before hal) agree on the error handling. |
|
yes, the error thing is a major pain with halui. |
|
A problem with TCP can be that it is not easy to notice if and when the other end of a connection goes away. Another problem can be connections that only transmit in one direction, without anything coming back, TCP delayed ACK can slow down connections to a crawl. E.g. Windows uses AFAIK 200ms per default, and you really are not supposed to change that. 200ms means 5 packets per second. It seems websocket does some message framing, that should take care of that. |
Network connections are notoriously difficult when you want guarantees. You'd have to start by examining the L1/L2 configuration. With Ethernet, it can take "ages" before you detect a disconnect. At least Tokenring had more real consistency. But then looking at L3/L4 with TCP/IP does not do much better because of routing and connection protocols. FWIW, TCP is just not good at timing, at all, and SCTP or UDP are a pain and no help when the underlying layers give you no guarantees. You will need to have a very well controlled local network to have some decorum. That leaves you with with everything (very) non-critical that can be done safely on a network connection. Anything that has intermediaries will be problematic with respect to timing and consistency. You only need to look at the efforts to get a point-to-point Ethernet connected I/O card stable (TCP/IP or Ethercat). And these are the well controlled environments. |
|
Remote HAL interface should enable to use something like halshow to run remotely. Remote HAL components with uncritical timing could also make sense to run remotely, things like a VFD interface. Anything "realtime" I would say doesn't need supporting and should be discouraged, at least over websockets. OTOH something like mesa hostmot network protocol or remora could possibly be extended to talk "hal" directly. |
|
Even a VFD interface can be time critical. Any operation that can "race" other operations must be properly and guaranteed sequence-able. |
|
Not sure what you mean, your typical VFD comms is RS485 at 9600bps, so in the context of a typical 1kHz servo loop, that is very slow and can't be time critical. Things should happen in lock-step, send command and wait for confirmation. Synchronised motion is for servos. |
|
chrome has a feature specifically for this: https://developer.chrome.com/docs/devtools/settings/throttling |
|
Some questions from reading the code, since a few assumptions in this thread deserve testing: On "zero copy matters locally": did you know the local SHMEM path already XDR-serializes every message? linuxcnc.nml says On the error problem: @BsAtHome, you said errors cannot be correlated across readers. Looking at the mechanism, it is worse: emcError is a queue with head/tail stored in the shared buffer itself, and read() destructively pops (cms_in.cc:817-828). No per-reader cursor, no retention. With two UIs attached, each error goes to exactly one of them, chosen by timing. Queue full and the writer drops silently (emctaskmain.cc:173-186). And motion/io faults mostly bypass the channel altogether; several emcOperatorError call sites are commented out (emctaskmain.cc:2801-2823). So the question is not "how do we serialize errors" but: what should the error model be? My sketch: task keeps a monotonic sequence-numbered ring, entries carry source (task/interp/motion/io), severity, and the originating command's serial; new entries are pushed to all subscribers, and a client sends its last-seen seq on connect to get replayed what it missed. Broadcast, correlation, and reconnect recovery from one mechanism, maybe 100 lines in task. What would it break? On scope: after halui moves into task, is the seam not just three files? shcom.cc (~60 functions; linuxcncrsh, schedrmt, emcsh all ride on it), emcmodule.cc (~40 command methods, ~88 stat attributes; every Python GUI), and emctaskmain.cc itself. The tool channel is already mmap (tooldata_mmap.cc), iocontrol already merged. Is anything else left that I missed? On the command handshake: today completion is a poll race on echo_serial_number with exponential backoff (shcom.cc:263-325; Python spins with esleep under a 5 s timeout, emcmodule.cc:922-960). If the transport is request/reply, the ack becomes explicit and carries an error reference. Any reason to preserve the echo mechanism at all? On transport: the rates involved are tiny (UIs poll status at 20 ms, errors at 200 ms, axis.py:789,834; nobody blocks on NML anywhere in-tree). Given that, is there an argument against running the same websocket server on loopback for local clients too, at least for v1? One code path, no parallel shm protocol; if profiling shows the status publish hurts, add a shm ring behind the same client API later, schema unchanged. On versioning: today the only skew check is the EMC_STAT_TYPE peek at connect (shcom.cc:90); emcmodule does a raw memcpy(sizeof(EMC_STAT)) plus an offsetof-based member table (emcmodule.cc:1022, :1124-1125). UI and task from different builds are silently undetected. FlatBuffers field evolution plus a handshake version fixes this for free. Worth stating as a hard requirement? On prior art: Machinekit already did this migration (machinetalk: protobuf over ZeroMQ, command/status/error/file services). Two lessons seem relevant: they kept NML alive beside it for years and the dual-stack dragged, which Rene's phasing avoids by shrinking NML to one seam first; and ZeroMQ never worked in browsers, so web UIs still needed a bridge, which websocket removes. Anyone here have firsthand machinetalk experience to confirm or correct that? And a starting point: rmu75's rs/zmq-experiments branch already has emc_cmd.fbs (583 lines), emc_stat.fbs, emc_error.fbs. The error schema there is a verbatim copy of the three NML operator messages, so it needs the seq/source/correlator treatment above, but the cmd/stat IDL looks like real groundwork. Rene, does it match what you had in mind for phase 3? |
|
Yes, machinekit never fully replaced nml, they did the classical Linuxcnc thing: putting things on top of other things, not removing the old one. I now have a branch with the first steps done, but will wait until bertos hal changes are done, as they do touch the same things in some places. |
|
On Sa, 2026-09-19 at 14:39 -0700, BsAtHome wrote:
> A problem with TCP can be that it is not easy to notice if and when
> the other end of a connection goes away.
Network connections are notoriously difficult when you want
guarantees.
We should not overthink this. Of course with TCP/IP you are on shaky
grounds in principle and there are no real guarantees in an IP network.
Token ring also had problems, it was slow, and, the token could get
lost, then somebody would have to generate a new one, after some
timeouts.
We just need to be prepared that a websocket-client can go away and
that it is not straight forward to detect that.
|
That is it. In the web it is usual practice to expect ping from websocket client every ... ms and if not received - close websocket server-side. Every public WS API working this way. |
can not it be done with ZMQ+flatbuffers instead ? it seems to be logical and easier. once it is done and proven to work, that is not a problem to make a bridge to websocket for those who interested. I would not want to control CNC machine via websocket... it is a little bit uncomfortable. ZMQ supports ipc:// transport protocol which sounds like an oblivious default setting for LinuxCNC. No TCP on this path... P.S. I am a web developer during all my professional life, and if necessary I am ready to create a sample implementation of websocket connection for this design + basic web GUI. Have done a really lot of those things. Another thought: current linuxcnc design requires a UI process to be running and linuxcnc dies when it is finished. So even if real GUI is remote/in-browser, some small daemon still have to be running on the CNC machine to pretend as UI. |
|
Im not talking about implementing websockets myself, there are libraries for that. we even already depend on them. |
There definitely are CNC machines that don't even need a GUI for normal day-to-day operation. Safety, ESTOP etc... has to work independent of any GUI anyways. I agree that typically you will have a "main GUI" running locally on the controller. Logic that reacts on non-responsive GUIs needs to be implemented separately but as rene pointed out nothing in task or anywhere else in the linuxcnc realtime part notices or reacts to a GUI becoming non-responsive. |
If you look at my zmq-experiments branch, I made a prototype / proof of concept replacement of CMS/NML with flatbuffers and 0mq (and no serialization for in-process messages). 0mq doesn't come without introducing its own set of problems though, you have to be careful not to indefinitely buffer messages for clients that become unresponsive and replay everything upon reconnection. Esp. if you "broadcast" things like status or error messages. |
If you can whip something up running in the browser and talking 0mq over websocket transport / flatbuffers like in my prototype, I can try to add 0mq websocket transport to my branch and look how it behaves. |
I just checked and it seems to be possible. Unfortunately very busy right now but will try to do it next week. |
|
0mq has (draft) websocket transport |
|
The focus on websockets as transport layer is wrong. The point of a good (generic) interface is that it is independent of the transport layer. |
It cannot work without going through both a network layer and a http layer.
You don't need an "extra" bridge if your interface is properly specified. You always need a primary bridge at the first interface level - from the machine to the abstract - to encapsulate the data. You also need a bridge at the end-point to make the data available in any sensible way. You are talking about additional bridges. You always need a bridge for each and every transport layer you want to support. Some need more work to handle data-representation properly at the end-point, others need less. The point is, some message passing systems support multiple transport layers in their library. That does not mean that you must choose only one. That would actually be bad, to limit the choice to one. In the case of LCNC, you need a distributed view of the status that is kept consistent at all times. Additionally you need bidirectional message passing, cooperation and synchronization between clients and between server/client. That can be achieved by any transport layer, but the upper layers that handle cooperation and consistency need to be discussed and developed. |
message queuing systems like zeromq already provide all that logic. |
No, they do not provide the consistency and cooperation we need for multiple LCNC clients. That is not what the messaging protocol solves. Some library parts can be used to map functionality, but it is still something we need to build on top of it. The application layer problems remain, even with 0mq or any other system (like NML). |
|
Of course the message queing doesn't solve the application layer problems, but it does solve keeping track of clients, that all clients receive broadcasts of subscribed topics like status updates and error messages, how to receive commands from different clients and return results, dealing with mangled messages etc... It won't prevent stupid like a user hitting CYLCE START on one GUI while another opens a different G-Code file at the same time on another client. |
It does not solve entire task but it solves may be 90% of the problem. Remaining 10% may be done in the dispatching code that calls ZMQ from linuxcnc "task" side. ZMQ also solves handling multiple transports and does it in very elegant way. |
|
I see it like this: we definitely want web browsers to be viable platforms for GUI-type or pendant-type applications, and that use case should be supported without too many hoops or bridges or whatever, else we get a similar mess like we have now where stuff like python -> JSON -> 0mq -> halui -> python -> deJSON -> halui -> NML -> task is proposed. From that requirement you can derive that some sort of websocket support will be necessary, because that is the only thing that is universally available from webbrowsers (apart from "naked" HTTP/AJAX, but we don't want that) -- correct me if I'm wrong. So now that we have to support websocket in one way or another, the question is, do we really need something else, and why. And we will need some sort of management layer that keeps track of all the connections/subscriptions/etc.... and some sort of RPC mechanism, and we really don't want to implement all of that ourselves. |
|
There is one good question.
Answer changes a lot because if the answer is [2] it simplifies a lot of things and makes task easier. In second case we can use Electron or Tauri and we can have a process that talks to LinuxCNC [ if necessary, but it is good to have ] and most important we are not restricted with usual restrictions applied for web pages [ which may be a real problem ] and can use some advanced optimizations / multi-threading . And if answer is [2] we actually do not need websockets at all, in this case raw TCP / ZMQ or whatever is here will be available. |
|
whatever people come up with, not necessary electron.
|
Sounds really good. I am in. I will try to try implement some ZMQ WS client-side next week and will report if it does not work. |
Just tried to build some proof-of-concept and you guys are correct. If [NML replacement] does not work as glue between linuxcnc core parts and [NML replacement] is only interface between linuxcnc core and UI, then websockets is the best choice and ZMQ seems to be a useless additional layer in between. |
|
yes, it used to be more, but I already removed a lot of it. |
|
There is a lot of talk that "websockets is the way to go", but there is no one who actually has described how the whole software stack is supposed to function. Please, explain, carefully and in detail, how the whole software stack is to be rewritten. How it communicates on every level, not just websockets, but the backend/task software stack and the frontend/VCP software stack. How is both consistency and synchronicity between clients and between clients and machine ensured. And how do you assure that the controlled machine not will go haywire when multiple clients are acting upon it. There is no point in fixating on any one single technology until we know what we are trying to solve. |
|
I have outlined a rough idea in my initial post, that replaces the same messages with flatbuffers, and sending them via a websocket. downside is, this fixes none of the problems, keeps everything as it is, including all the bugs. upside is, none of the UI code needs to change. Nobody would ever know it isnt NML below. Linuxcnc has many UIs, and not all have maintainers. We probably want more than that, but it requires massive changes to the way UIs work, and that is exactly what this discussion is for, to get input about what we need to think about. Additionally the question Im asking, is there anything that Im missing when using websockets. Anything that really speaks against it. Im also looking for input on how to handle hal, and how to handle the preview. I outlined some preview problems below. |
The point and question I am making is: Is there anything the speaks for it to make it the exclusive topic of conversation. To me that is a very bad sign, to decide the technology before knowing the actual problem. Until now there only has been talk about "we use this" or "we use that" without addressing the actual problem. All these "use this, use that" statements are useless as long as we do not address and specify the actual problem. |
Ok, let me describe the actual problem as I see it. I have made several [ some were successful ] attempts to write an UI for LinuxCNC. It is very hard to do, there are many problems that need to be solved on this way. In fact only the choice now is to use Python with bundled libraries "as is" if you are not going to invent a lot of wheels. I dont blame I understand it is result of long evolution of LinuxCNC code for long years. However it is time to make LinuxCNC better adopted for rapid UI development and once it is done, there will be good results. |
|
Unfortunately I won't be able to attend the meeting on Saturday, so I'll describe my vision here instead. I've done some experiments implementing a web UI on top of the existing code, and it's almost finished, so I have some experience with what needs to be done and how to bring it to life. It was exactly FlatBuffers over WebSockets, with the browser client written in React/TypeScript, so I can confirm this combination works and is usable. Here is what I found. Exchange pieces required for integrationDirection is from the LinuxCNC side. I describe here what will be necessary for Web UI development, because it requires the maximum amount of integration. Legend: Well-known / mirror of NML data
Specific to the new UI / web app
P.S. Forgot one
Conversion of existing clientsExisting clients all use the Python interface because there is no other choice. Once it is rewritten to use WebSocket, they will simply work as before. |
Not a requirement, but it would be good to have both in one process without special tricks. Right now as you definitely know well, it is constructed on linking stage. I see no reasons why it must be done this way, I believe modern C++ allows to do the same in more natural way and remove that restrictions [ only one interpreter in process ] . We need at least 2 : first to run program actually , second to render preview / check extents. If there are any showstoppers for it, please advise, I am not familiar with this part of code. |
|
Your example is not two parallel interpreters as you can use one interpreter twice, like is done today. The question is where and when you actually want two parallel interpreters running concurrently doing different things? What is its use case? And, running one to make the preview and one to run the program do not run concurrently. You wouldn't want the program start cutting away while you are still waiting for a preview (if you have enabled a preview). Secondly, with concurrent clients hooking onto one machine, you do not want to re-run the preview generation as this is a serious synchronization bottleneck. |
If I'm not mistaken this could be a use case for that. |
That is EXACTLY two parallel interpreters! When preview generated it plays every move in the program. You do not want actual axes to be moving in same time, do you?
May be you did not check this part before or just did not mention this detail. Right now you simply cannot have in one linked binary interpreter that move axes and an interpreter that renders preview. It is impossible. |
The CANON Api is statically linked. You are limited to one running interpreter per process and even then you can't easily retarget who it talks to. Adding a multiplexing-layer on top of CANON interface is wrong imo, cleaner solution would be to make interpreter multi-instance clean (move global state into per-instance state) and give it some object implementing the equivalent of current canon interface. I don't see the need to make this interface available over process or even machine boundaries, so plain C++ would suffice IMO. |
|
The NML and associated code more and more proves to be a burden in the linuxcnc project. The upstream libnml seems to be more or less unmaintained, the protocol seems to only have C++ bindings (also using a quite old C++ pre 2000 style) and lacks a kind of IDL and associated code generation. People feel uncomfortable changing CMS structures or implementing new services with NML and resort to "bolting on" parallel communications facilities (see the recent zmq/halui controversy, even motion state tags is an example of hesitation to properly extend CMS structs). In other words, the system is obsolete and the sensible way forward seems to be to replace all that with something that is understood, maintained and supports modern languages. As demonstrated in a few experiments, replacing the CMS structures and NML with something else, flatbuffers or even bare C++, is not the hard part, as far as only in-process communication is concerned. Even 1:1 port of current functionality to something like websocket or zmq is more or less straightforward and prototypes exist. The motivation to do all that is to get a new future proof system that solves painpoints like errors currently going only to one consumer (an arbitrary one if there are more listeners of the error channel), that is easier to extend and maintain, and that supports new features and avoids "putting things on top of other things" and creating parallel infrastructure. So, as a long-term goal, vision and IMHO, the system should support multiple GUIs running on networked computers and even in a webbrowser. GUI may be something like gmoccapy or something lightweight like just a DRO or a typescript application on your phone that allows you to use it as a virtual handwheel. Multiple GUIs that can connect/disconnect at arbitrary times means the GUIs should not carry any machine-state. We want to avoid things like one GUI talking to another over ad-hoc mechanisms to change state there (like a jog rate). Such state should either be kept centrally in task or in a separate component that lives independently of the GUI. The current approach of generating the previews in the GUI process is not without problems, there are cases where the main interpreter is desynchronised from the GUI and surprising things can happen with regards to machine limits, offsets etc... A new RPC system should allow for efficient transport of previews of G-Code and backplot of machine movements. (There is also some proof of concept of preview generation in task floating around somewere)
|
|
We cannot decide the technology to use in step one. |
|
OK i will split point 1 |
|
I sketched out the problems I perceive we have and should be solved, please amend if I'm missing something or you think something is out of scope. |
I think this is enough of a problem statement. We should choose a solution that supports forseeable requirements, doesn't impose arbitrary limits, is maintained, documented, has mind-share and won't go away in the near future. Any solution chosen is not permant, it could be changed again if some fundamental problem would show up. I don't think it makes sense to plan in all detail everything in advance. After all this is not a nuclear power plant, the code is evolving, what works remains and is extended, what is problematic will go away, like NML now. |
|
Some possible comm frameworks (please add)
Serialization frameworks:
|
|
There is one argument against websockets is that it involves TCP/IP stack. It is not necessary true. libwebsockets (libwebsocket-dev debian) provides both server and client bindings, and it supports TCP, TLS (wss://), HTTP/1.1, HTTP/2 (WebSockets over h2), raw sockets, adopting an existing fd (serial, pipe, socketpair). Can run on its own poll loop or on glib, libuv, libev or libevent. The glib option lets it share a GTK GUI's loop [ good choice for UI integration ] So websockets here does not mean linuxcnc go to networking, this protocol only defines handshake and messages framing and it is well supported anywhere. I agree that default setup should use AF_UNIX sockets and not TCP networking to avoid security risks. |
|
Am Mittwoch, dem 07.10.2026 um 23:52 -0700 schrieb BsAtHome:
> currently you cant instantiate 2 interpreters.
Is this a requirement?
It has nothing to do with the NML discussion.
|
Well, it might be if the client is to become a dumb frontend. Then there may actually be something that needs to be taken into account how to hook onto what. Ruling it out categorically may not be prudent at this stage. |
|
If the client becomes a dumb frontend it will have to deal with the interpreter even less than today. |
|
I'd like to keep one more note because I am unfortunately unable to attend meeting, but I hope it can be considered in planning. It is important to separate WS streams (or whatever there will be ) to at least 2 channels [for WS it could be 2 URLS]
Another one that could be good to have is ability to stream NGC file contents. Browser will hang if it get megabytes of text at once, and it is usually not necessary. Server should reply to any client request to read NGC file , however in usual mode it would be nice to have such protocol:
|
|
Yes, that is something to consider. Gcode handling of remote UIs is definitely something to think about. Note that neither side of nml is realtime, and a browser is not necessarily involved at all. Even if they are, a browser is perfectly capable of uploading gigabytes of data while doing something else, webworkers can be multi threaded. Currently the UI is also stuck while loading the file, as the preview runs in the ui thread. |
|
Exactly, it is better to get fixed. I say "browser" but of course it applies to other UI, browser is just most limited one so it is the best example. If it works in browser, it will definitely work anywhere else. And of course it is not really realtime, I mean here "data updating fast". |
|
Another one. NGC handling on remote UI [ and even on the same PC via browser ] is a classic job for a HTTP server. Or even better, WEBDAV server can handle it and it is widely supported in software. |
|
This is again choosing technology before the problem is clear and analysed. It is completely backwards. How about I tell you to require using one specific material, lets say, sand, and you have to build a 12 story house out of that. |
The problems is clear [ for me ] and analysed [ by me ] . I had a lot of time to think about all this when I developed web UI for LinuxCNC, C++ GTK UI and Python QT UI. That is why I keep this note for information in hope it will be useful [as only my opinion and nothing more]. If you have other ideas, it would be interesting to hear.
That is a good example. If you ask one who built several similar houses he already knows what material is good, right? |
|
When I added hal_port pins for laser rastering data... the thought of replacing NML shared memory communication with port pins crossed my mind.
I envisioned using a text based protocol for easy debugging to store commands in the trajectory planner. The trajectory planner would then have functionality to execute commands, store motion points and respond via an output port much like a terminal application. It would also be able to report status and logs via that port.
I think web ui would be the next layer up and communicate to the USPACE side via some sort of standard sockets.
Thanks,
Curt
…On Fri, Oct 9, 2026, at 10:43 AM, BsAtHome wrote:
This is again choosing technology before the problem is clear and analysed. It is completely backwards.
How about I tell you to require using one specific material, lets say, sand, and you have to build a 12 story house out of that.
—
Reply to this email directly, view it on GitHub <#4571?email_source=notifications&email_token=AACTC4YW2HUV4JATC7WWFJ35TD2RZA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBYGM3TINJQUZZGKYLTN5XKU43VMJZWG4TJMJSWJJLFOZSW45FMMZXW65DFOJPWG3DJMNVQ#discussioncomment-18837450>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACTC44ALJUI2ULFRMIDH5T5TD2RZAVCNFSNUABGKJSXA33TNF2G64TZHMZTMNRSHEYDKO2ENFZWG5LTONUW63R3GEYDQNBXGEZDHILWAI>.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AACTC45HYLVSBLIMQYCGXML5TD2RZA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBYGM3TINJQUZZGKYLTN5XKU43VMJZWG4TJMJSWJJLFOZSW45FKMZXW65DFOJPWS33T> and Android <https://github.com/notifications/mobile/android/AACTC47I4G4VTYR4SZW2I7T5TD2RZA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBYGM3TINJQUZZGKYLTN5XKU43VMJZWG4TJMJSWJJLFOZSW45FOMZXW65DFOJPWC3TEOJXWSZA>. Download it today!
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
|
Am Freitag, dem 09.10.2026 um 07:43 -0700 schrieb BsAtHome:
This is again choosing technology before the problem is clear and
analysed. It is completely backwards.
How about I tell you to require using one specific material, lets
say, sand, and you have to build a 12 story house out of that.
Thats not a very good example. It is obvious that with sand alone you
can't build that house, but you could if you add cement steel and
water.
Also it is obvious that websockets can be used to implement remote GUIs
and be used as transport for every other imaginable remote procedure
call that could occur. The questions are *) does it block something
that might come up in the future and *) are there better solutions,
with the additional problem that it is not obvious what "better" means
in this context.
|
No, you don't need cement, steel or water. You just need to squeeze the sand into a mountain (as nature does to create sandstone) and then remove everything not needed. This example is showing you that by fixating on one way to solve things that you become blind to alternatives. |
|
Before Sunday, a sketch of an architecture that could address the current issues, to be refined with community input; a first attempt at the architectural base @BsAtHome keeps asking for. The transport question comes after this. @c-morley's problems in #4613 are real: halui and every GUI keep their own jog rate, axis selection and increment, and a panel's cycle start has to let the GUI do its pre-checks. What is missing is a model of where each kind of state lives and who may act on which input. #4613, my #4616, #4653 and #4644 all move code between processes without one. Three kinds of state
Two kinds of input
#4613 mixes the two: broadcast to every GUI, two running GUIs both react to the same OK. Focus and arbitration With several UIs running, one must be the one in use. A UI claims focus explicitly, the core arbitrates, and screen inputs go to the UI holding focus. Machine commands need arbitration as well: today there is one mode and whoever switches it last wins, which is where the mode-switch races come from. Run names the program it expects One copy of the state is not enough. Operator A sees program X, another UI loads program Y, A presses cycle start, Y runs. A run command should name the program it expects, and the core refuses if a different one is loaded. A panel cycle start names nothing itself, but it still goes through the request and answer step below: the UI holding focus can confirm what is about to run and say which program it shows, so a program changed from another UI is refused there too. Pre-checks
The GUI no longer intercepts buttons; the core stays the arbiter. One rule Feed hold, stop and abort never depend on a GUI being alive. What I would ask the meeting to agree
With that, the #4613 pins split cleanly (machine commands to the core, screen inputs to the focused UI), #4644 has a clear target, and the NML replacement has requirements to be measured against, whatever the transport. Edit: narrowed the revision idea to the run command, after @rmu75's comment below. |
|
One more thing the request and answer step would give us almost for free: interactive G-code. A program asks the operator a question with a few choices, the UI in focus shows it, and the answer lands in a parameter the program can branch on with O-word if. Today that needs fixed INI dialogs wired through HAL and read back with M66, one GUI at a time. |
|
Architecture looks good! One note, as I said above it would be cool to have [3 - View state: page, tab, preview rotation, open dialogs. Belongs to each UI.] also optionally stored on server if UI desires. It is not expected that linuxcnc understand it or know format, UI will simply use it like backend to keep data between browser UI reload/ desktop UI restart. |
|
State being kept in the UI should be avoided. IMO firing an event like "workoffsets have changed to G55" would be a mistake, only complete machine-state should be transmitted, and edges / events should be derived in the GUI if needed. Thus synchronisation issues are completely avoided. Stuff that is too big to transfer with each update like the g-code of the currently running program, the preview/backplot and the tooldb needs to be cached on the client-side. Re. tool db we should also have a closer look at the stuff done by Brad Collette / FreeCAD https://github.com/loobric/loobric-freecad Adding a generation number to state to try to prevent stupid races is fruitless IMO and just adds complexity for little benefit and may even lead to "surprising" behaviour. Two operators working on two stations affecting one machine where one loads a program while the other hits cycle start is not a problem of the technology. In an industrial setting, you would need interlocks and whatever anyways. Apart from that, I don't see how this discussion helps in deciding what comms technology to use because comm patterns involved seem to be exactly like they are now with NML and seem to be pretty ordinary: central server, clients connecting to server, server "broadcasting" status, either continously or per request, and clients doing request/response stuff. Any comms layer will support that much. |
|
Agreed on state: the UIs keep only view state, and full state broadcast with edges derived in the GUI works for machine and operating state alike. On revisions, fair point, a generation number on every piece of state is more machinery than the problem deserves. I narrowed it in the sketch to one case: a run command names the program it expects, and the core refuses if a different one is loaded. It is not only two stations either; a pendant or a browser next to the main GUI, or a client that reloads the program by itself, is enough. On comms, mostly agreed, the patterns are ordinary. If I am not wrong though, NML is missing at least one pattern the sketch needs: the core asking one particular client something and waiting for its answer, like the confirm step before a cycle start. As far as I can tell it only goes the other way, clients sending commands and everyone reading the status, and command completion and errors are not tracked per client either. I would want whatever replaces NML to cover that from the start. I did think about slimming NML down and modernizing it instead, but I don't think it is worth the effort. |
|
Following the meeting (conclusions: prototype the replacement and see what has to change, evaluate ZMQ over websocket), three things for the prototype. In his comment on #4613 Chris listed what makes GUI work painful today. These are not requirements for the sketch, but things worth moving towards fixing, and the new layer is the chance to do it. Where the sketch stands on each:
View state stored by the core Agreed with @alex-pres: the core can keep view state for a UI as an opaque blob it never interprets, so a browser reload or a GUI restart picks up where it left off. I will add that to the sketch. Encoding: JSON with deltas as an alternative to flatbuffers A colleague suggested JSON plus a protocol that only sends what changed:
Since the prototype is there, the fair way to decide is to measure both encodings on the same messages, |
|
Some preliminary numbers on the "JSON, send only what changed" idea, measured against rene's websocket-wip branch. Setup: rene's branch as is, with a hook in its websocket server that encodes the status every task cycle (1 ms) while a sim runs. Two configs: the axis 3-axis sim running 3D_Chips.ngc, and the 9-axis sim running a program that moves all nine axes at once, as a worst case. The JSON delta is a merge patch (RFC 7386) against the previous status, with joints, axes and spindles keyed by index so only the entries that changed are sent. Size per update:
An idle machine sends about 27 bytes per update, since only the heartbeats change. Encode time per update, from EMC_STAT inside task's server thread:
The changes-only encoder compares each field with the previous status and writes text directly, so it is cheaper than building the full FlatBuffer. Replaying the deltas onto a snapshot reproduces the full status exactly. Decode time per update:
The last row uses simdjson with a decoder generated from the .fbs, writing each changed field straight into the struct. Replaying the deltas that way also reproduces the full status exactly. For a pure Python client talking to the socket itself (3 axes): unpacking a full FlatBuffers status takes 509 µs and reading just 5 fields 18.5 µs, because the Python FlatBuffers runtime is plain Python; parsing and merging a JSON delta takes 5.6 µs with the standard library. GUIs going through the My reading: on size and encode cost, changes-only JSON is at least as good as FlatBuffers for status, and with a generated decoder it is within a few µs of FlatBuffers on decode too, small next to a GUI refresh period. Pure Python and browser clients get JSON from their standard library. The schema does not have to go away either: both the JSON encoder and decoder in the offline test were generated from the same .fbs files. What it costs, and what I have not done yet:
The code is on my fork, rene's branch plus one commit: https://github.com/grandixximo/linuxcnc/tree/nml-json-proto. |
|
@alex-pres agreed, and thanks. Read in place, FlatBuffers decode is cheaper than anything JSON can do, and my "at least as good" went too far on that side. The size win in my numbers comes from sending only what changed, not from JSON, and your sections get it while keeping FlatBuffers. Part of the 3.5 KB is cheap to cut regardless: the 64 analog in/out doubles and the 64 digital in/out and misc error entries are about 1.4 KB of every frame and almost never change, so they belong in a section of their own. So I drop JSON as the main encoding. What I would keep from it: deltas with a per-connection sequence and full resync for status, as in your Edit: @alex-pres's two comments I was answering are no longer in the thread. In short: decoding FlatBuffers in place, for example from a C++ Python module, needs no unpack, so it beats JSON; and status can be split into optional sections (task, program, motion, axes, IO, ...) sent only when they change, with a per-connection |
|
Sorry removed my comments because I re-read your message carefully [finally :) ] and there is nothing to argue. Thanks for accepting the design idea, it is proven to work. I can re-paste it again if useful. enum StatusSection : uint (bit_flags) {
Task, Program, Modal, Motion, Axes, Spindles, Offsets, Eoffset, Io, Hal
}
/// Per-axis readout. Positions in machine units.
struct AxisStatus {
commanded: double;
feedback: double;
/// commanded minus every offset in force (what a DRO shows).
relative: double;
velocity: double;
homed: bool;
on_min_limit: bool;
on_max_limit: bool;
}
struct SpindleStatus {
enabled: bool;
direction: SpindleDirection;
/// Commanded speed as reported by motion (zero when off).
speed: double;
override: double;
brake: bool;
homed: bool;
}
....
/// The machine. Pushed at `ServerInfo.poll_ms`; whole, or in sections when
/// the client asked for deltas.
table Status {
/// Monotonic per connection, counting frames *built for this client*.
/// A tick the server suppressed because nothing changed does not advance
/// it, so an idle machine leaves no gap. A jump does mean frames were
/// dropped for you: the server shed queued frames because this connection
/// was not draining them, and the frame carrying the jump is `full` for
/// that reason. Reading the two together is the whole rule — a gap with
/// `full` set is a resynchronisation, and there is no other kind.
seq: uint;
/// Server monotonic clock, milliseconds, at the poll that produced this
/// frame. `uint` rather than `ulong`: it wraps every 49 days, every use is
/// a difference between two nearby values (correlating an event with the
/// frame it fell beside, measuring socket lag, spotting a stalled poll
/// clock), and a 64-bit field arrives in TS as a `bigint`.
t_ms: uint;
/// Every known section is present: take this frame as the whole truth and
/// discard what you held. Always true without `Hello.deltas`; with it,
/// true on the first frame after `Welcome` and whenever the server
/// resynchronises. A client needs no other rule to know it is in step.
full: bool;
/// Sections this server answers at all. A section outside it is absent
/// from every frame and carries no readings, ever. A property of the
/// server rather than of the frame, sent on every frame because four
/// bytes cost less than a rule about when it was last stated.
known: StatusSection;
/// Sections whose readings differ from the frame **before this one, on
/// this connection** — always masked to `known`, so a section the server
/// cannot answer never appears here.
///
/// On a delta frame this says exactly what presence says, and it is worth
/// having anyway: it is one field to test instead of ten null checks, it
/// is `0` on a heartbeat (nothing moved, the server is alive), and both
/// are written from the *same* comparison in the server, so they cannot
/// come to disagree.
///
/// On a **whole** frame it is the only thing that says it: every section
/// is present every time, so presence carries no information there. That
/// is what this field is for — a client can take whole frames, which need
/// no merging and no accumulated state, and still repaint only what
/// moved.
///
/// On the first frame of a connection there is no frame before it, so
/// everything differs and this equals `known`.
///
/// It is **section**-granular and deliberately not field-granular: the
/// sections are grouped by what moves together, so a bit per field would
/// be sixty-odd bits restating what ten already say, maintained by hand
/// against every field ever added.
changed: StatusSection;
task: TaskSection;
program: ProgramSection;
modal: ModalSection;
motion: MotionSection;
/// Indexed as `MachineInfo.axes`.
axes: [AxisStatus];
spindles: [SpindleStatus];
offsets: Offsets;
/// External offsets (eoffset) in force, per axis, machine units. Its own
/// section rather than a field of `Offsets`: while taper compensation is
/// active this moves every frame and the work offsets do not.
eoffset: [double];
io: IoSection;
/// Values of the pins named by the last `HalWatch`, in that order.
/// Sampled on the same poll as the rest of the frame.
watched: [HalValue];
}
[ fields in FB are usually optional and may be omitted from the message ] |
|
Well up to you guys, I just never used flatbuffers, and I think JSON is native everywhere, so it has its perks... |
I love JSON, but I believe LinuxCNC must [ and easily can at this point ] be better optimised to run on some RPi/etc without hassle. However it would be good option to send JSON [ and may be the same format as you suggested - JSON delta ] from server-side if client opt-in. It is easy to implement and for web UI it is an easiest option. |
|
I don't have time to dig into it right now, but I'm pretty sure flatbuffers supports serializing/deserializing flatbuffers structures to JSON, originally as a debug tool. https://flatbuffers.dev/benchmarks/ contains some numbers. |
|
The current suggested change (disregarding the transport) would be something like this: |
|
Following up on "prototype it and see what has to change": a sidecar sketch that needs no core change, https://github.com/SyncTwin/lcnc-state-layer. It is a plain NML client (the stock Numbers on stock master 2655e62 (axis sim, axis.ngc, all 91
This agrees with @grandixximo and @alex-pres: size comes from the data model, not the format. With stat's fixed 16-entry Also run once on real servos (EtherCAT, three drives on X/Y/Z, jog ±2 mm through the sidecar, 100 ms): the streamed position matches the drive encoder within one tick, 0 seq gaps, sidecar 1.2–1.4 % of one core with one client. Two findings: encoder noise (±4 counts) changes positions every tick at rest, so without a deadband there are no empty ticks; and Not done: operating state / halui sync (Chris's first point), several clients sending commands at once, CPU with N clients, flatbuffers' own JSON output (@rmu75's point). |
|
Same bench on a Raspberry Pi 5, since RPi came up above. Setup: Pi 5 (Cortex-A76, 4 cores), Debian 12, kernel 6.12.11 PREEMPT_RT with isolcpus=2,3. Bench and sidecar run on the non-isolated cores 0-1. LinuxCNC 2.9.4 sim (axis_mm, headless), axis.ngc in a loop, 100 ms poll.
servo-thread jitter was not measured here. Raw numbers and env: https://github.com/SyncTwin/lcnc-state-layer/tree/main-public/bench/results/rpi5 |
|
@yurc, very interesting. 2 ALL: one thing I would propose into the protocol: report number of joints and axes In Hello message from server, and only pass this number of structures. It is not good to pass 16 joint status while working on a 2-joint-axis lathe, it is just waste of resources. Also it will allow to increase number of joins ant axes in future if necessary. |
|
@alex-pres agreed, done in the sidecar: the server now sends a What the sidecar cannot solve without core changes (protocol version at connect, a single reader of the error channel shared with the GUI, …) is listed in CORE_CHANGES.adoc — the sidecar is meant as step one; the same protocol could later be served by LinuxCNC itself. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
During the Stuttgart meeting, I gave a NML replacement some thought.
The only positive thing to say about NML is that it has served us well over the years.
I think we all agree that it is unmaintainable, and nobody wants to touch it.
In the past, and probably even more likely in the future, people always wanted to have/create remote UIs, especially web based UIs.
While this is in theory possible with nml, there are very few examples showing how to do it, and web UIs require a bridge.
there have been experiments with zmq and flatbuffers, e.g. https://github.com/rmu75/linuxcnc/tree/rs/zmq-experiments
and flatbuffers actually seem like a good idea. they are zero copy, efficient, and bindings can be generated for almost any language. Leaves the question of transport and framing. websockets do both, and work native in a browser, and almost any language supports websockets. It is mostly just a tcp socket that has been set up in a special way.
so here is my plan:
1 remove nml where it is used only for helpers
this are mostly timers, some types, and other little helpers. I am currently working on this, it can go straight into master, as it doesn't really change much.
2 move halui into task
halui talks to task via nml, and mostly exposes its state machine to hal. there is no reason task cant do this directly.
I have done the same with iocontrol, and it saved over 2000 lines of code, and exposed a lot of race conditions.
#2497
this can also go in master, as it doesnt affect anything user facing.
after this, nml is only used in communication between task and the UI.
3 new c++ api
create new c++ api, that talks via flatbuffers over a websocket to task, with the same messages as nml.
this wil be used by the c++ clients, used mostly for testing. this is what shcom.cc currently does, but extended so all UIs can use it.
4 create python bindings for the c++ interface
this is what emcmodule.cc currently does. In theory, no change to UI code will be required. Practice will probably tell that there are quirks of nml addressed in the UI.
as a stack, I suggest using beast, which is part of boost, which we already depend on.
Preview
currently the preview interpreter instance lives in the UI, which means every UI has to load it, it needs bindings, and every UI needs to manage it. I had the idea to move it to task, so that the UI can request parsed chunks of gcode, and only needs to render it. it has the advantage that limits are applied in headless systems, and it massively simplifies UI code.this is a big change for the future, and needs changes to the canon interface. just dropping the idea here.
HAL
currently a remote UI doesn't have access to hal. I am still thinking about how to solve this in a way that the same c++/python api can be used via SHM or network transport.
All reactions