Feature/mtconnect/agent - #4318
Conversation
3e509ee to
df19f2f
Compare
|
Moving back to draft pending current (and future feedback). |
7a2952c to
ff16948
Compare
Add a first-class, INI-configurable MTConnect feature: a userspace, non-realtime HAL component that reads machine status, kinematics and tool data via the linuxcnc Python module and publishes them over MTConnect, with no dependency on the external cppagent. Transports (INI [MTCONNECT]TRANSPORT = http, mqtt, shdr): - Embedded HTTP agent: /probe /current /sample /assets, plus /twin and /models. - Standard MTConnect MQTT binding (retained Probe = discovery); optional, python3-paho-mqtt as a Recommends. - SHDR adapter (default port 7878) to feed an external agent such as cppagent, which is configured with a Devices.xml produced by --dump-probe. Device model, generated automatically from [TRAJ]/[KINS]/[AXIS_*]/[JOINT_*]: - Axes as Linear/Rotary components with Motion elements and CoordinateSystems; a compact x:Kinematics extension carries the kins module, coordinates and the joint<->axis map for auto-configuration. - Travel limits and the [SPINDLE_0] speed band as Specifications; work offsets (G54..G59.3 + G92) and tool offsets as WORK_OFFSET/TOOL_OFFSET tables; tool table as CuttingTool assets; coolant flood/mist; XY work rotation. - LINE_NUMBER reports the executing line (stat.motion_line), not interp read-ahead. - Arbitrary HAL pins as data items via [MTCONNECT]HAL_ITEM (hal.get_value). All lengths are emitted in canonical millimetres with nativeUnits. Documents validate against the MTConnect 1.7 schemas; a small LinuxCNC extension schema (mtconnect-linuxcnc-1.xsd) covers the few LinuxCNC-specific items that Streams cannot otherwise represent. Digital twin: an opt-in browser viewer (ENABLE_TWIN) that loads geometry referenced by SolidModel and animates it from /current; MODEL_AUTO generates placeholder geometry from the travel limits. Its three.js comes from the distribution's libjs-three package (served from /three/), so no third-party JavaScript is vendored into LinuxCNC and the twin still works offline. Packaged as a userspace component: src/hal/user_comps/mtconnect-agent.py (USER_COMP_PY -> bin/mtconnect-agent), the mtc package under lib/python, and data assets under share/linuxcnc/mtconnect. Offline test suite in the demo config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Man page docs/src/man/man1/mtconnect-agent.1.adoc (auto-globbed into the build). - Integrator chapter docs/src/config/mtconnect.adoc, registered in Master_Documentation.adoc and the docs Submakefile. - Document the [MTCONNECT] section in ini-config.adoc. - Demo-config README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Home Assistant's MQTT Discovery is an HA-specific convention, not an open standard, so it is deliberately kept out of the core agent. This optional, opt-in contrib republishes the agent's device model as HA discovery. - configs/sim/axis/mtconnect/contrib/mtconnect-ha-bridge: reuses the agent's device model to publish HA discovery + state, configured entirely with arguments so no HA-specific key is placed in the machine INI. - configs/sim/axis/mtconnect/contrib/ha.py: the discovery/state helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the bundled pure-Python INI fallback from IniReader and always use linuxcnc.ini(). The fallback was a prototype convenience for running offline without the linuxcnc module, but now that the agent ships inside LinuxCNC that module is always available -- and the fallback parsed differently from linuxcnc.ini (it stripped inline '#' comments, whereas LinuxCNC keeps everything after '='), which could mask configuration bugs. IniReader is now a thin typed wrapper over the canonical parser. Raised in PR review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ff16948 to
523fa38
Compare
|
I've removed the vendored threejs in favor of the older debian package. |
|
I'm not sure we should make this a requirement for LinuxCNC. The mtconnect is an addition to the system not really necessary for running a machine. It is actually a very specific addition for a very select group of users. Adding hard dependencies pulls in a lot that, IMO, should be no hard requirement. Don't get me wrong, I think this project certainly has merit, but I'm concerned it only has very narrow application. Certainly something you'd want to choose to add, but a requirement for a normal install is a bit much. At least it should be separated out of the main .deb. I'd like to hear other's opinion on this subject too. |
I think this is needed. Maybe not today, but tomorrow. There have been a number of attempts at http: interfaces etc but none of them are standards based and none of them address security issues satisfactorilly. In fact, I recently proposed a secure solution which a friend James Walker in NZ coded. https://github.com/joco-nz/linuxcnc-Headless-UI MTconnect is standards based and is supported by all of the major manufacturers and Siemens provides native support in their controllers. Being read only, security is not an issue the standard needs to address. Fanuc has been forced to offer a MTconnect adapter despite having its own proprietory communication system. LinuxCNC should not ignore this standard as it could preclude its inclusion in a factory fleet, perhaps where a bespoke machine is needed. I personally know one workshop owner running 7 LinuxCNC machines which are all retrofitted "big iron" machines. He could benefit from data collection and a centralised view of his workshop. At a basic level, Home Assistant via MQTT is seamless and might suffice in that scenario. There have been an increasing number of machine manufacturers adopting LinuxCNC that have a family of machines to address a vertical market, be it a stone manufacturer in China or Wood machinery in India (2 examples known to me) that could benefit from tomorrow's technology if available today. In one case study, an automotive plant avoided downtime that cost $1.7m an hour because their MTconnect data identified increasing torque on a single motor on a CNC axis and could schedule repairs without downtime. Why can't LinuxCNC earn a place on that factory floor? It can't without MTconnect. To me, adding optional MTconnect support via a couple of optional INI parameters is a no brainer and I strongly support this PR. To wave it off as being a very narrow application is simply out of touch with the reality of modern CNC machine applications. We should not be living in the past. |
A lot hinges on the word 'this'. Do you mean digital twins or MTConnect in general? I can take or leave digital twins. They are nice and flashy but not critical. MTConnect as a supported standard, OTOH, is probably more important than you think and will be more important going forward. I've been paying attention to it for five or six years and only recently realized how important it might become for integrating a machine tool into a broader ecosystem of monitoring, CAM, dashboard, and related tools How this is supported is really the crux of the PR and the single most important question. Let me illustrate.... It's the year 2028 and someone googles (or asks chatgpt) "Does LinuxCNC support MTConnect?" Do we want the answer to be ... or B) I wouldn't argue the same way for every feature under the sun. But if the feature is about supporting publicly available, verifiable standards, then I think the development community should strive to make the experience seamless, predictable, and robust. That makes it easy for content creators to make videos. It makes it easy for commercial companies to choose linuxcnc as an embedded control system. And it makes it easy for hobbyists to tinker. |
|
For the record, Sliptonic's pre-release work has already been implemented in the riocore project which supports linuxCNC on unconventional hardware. https://github.com/multigcs/riocore/tree/dev/riocore/plugins/mtconnect |
|
If MTConnect support is what this PR is about, I think the twin and libjs-three could be dropped. Better alternatives already exist: the cppagent repo ships a generic three.js twin (demo/twin, what demo.mtconnect.org/twin runs) that renders any agent from /probe alone, WebXR included, and TrakHound/Ladder99 cover the dashboard side. SolidModel geometry is referenced, never streamed, precisely so external viewers do this job. If a native built-in viewer is wanted, I would rather see something JS-free (a server-side SVG projection of the MODEL_AUTO geometry would be dependency-free and air-gap safe), but I am not putting my foot down on it; if there is a strong argument to keep it as is, I am happy to hear it. Dropping the twin also drops the dependency story to a single, essential, much more palatable item: python3-paho-mqtt, ~100 KB, only needed when the MQTT transport is used. One thing worth considering for the native-vs-separate question: release cycles. In-tree means agreeing to a slow rollout (2.10 is far out, distro packages lag further), while a separate repo with a self-managed apt gets fixes to users in days. The agent has zero core coupling, so both models are viable; just be aware of what you are signing up for either way. |
|
I really like the project and the integration of a simple front end. On Linux in particular, it’s not easy to find something that works straight away, and not everyone fancies setting up a database and Grafana just to have a look at it. However, I’m also of the opinion that it would be better off in its own repository. After all, it’s a standalone component that’s much easier to maintain that way. |
Note we already have user space components that require this but we have not added it as a dependency.
Its not a stand alone component, its part of a modern machine tool's infrastructure. MTconnect is a mature open source standard with a defined scope supported by most major machine manufacturers. This means there will be fewer changes coming than you first think as the initial feature set is very complete. If you leave it for a third party to host, nobody will find it. LinuxCNC should position itself for the future and include these few lines of python code. |
|
This project, as I said before, has merit and do think it to be a good project. But, there are some issues we need to resolve. I think it is a very nice add-on project/component and should be be handled as a separate deb package. I'd even argue that many more parts of LCNC need to be split into separate deb packaged and this project just reminded me to bring that up. That doesn't mean it can't be in the repo, or under LCNC umbrella, but we need to start modularizing. There are questions about the viewer part. Adding all the files from three.js to the repo will become a maintenance nightmare and they aren't even part of the core project. Sure, having the viewer is a nice thing to have, but shouldn't that simply be pluggable on the user's wishes? Also seen in the light that there may be alternatives, a pluggable system would seem better and it would be a separate project. It may be under the same umbrella or separate. Adding MQTT is not the problem. Adding MTConnect is not the problem. |
MTConnect being stable and unchanging is not really the issue here. What we are talking about is the glue between LinuxCNC and MTConnect, and that glue is brand new. I already found a few real issues in it (see my review comments above), and as Bertho pointed out, things change in LinuxCNC that will affect the glue over time. The standard being mature does not make the integration mature. That said, I don't think this "small" python add-on should be split out on its own. There are plenty of optional, narrow-audience comps already in the main deb (mb2hal, mqtt-publisher, the VFD comps), and starting the modularization experiment with the newest arrival would not be consistent. I agree with Bertho that more of LinuxCNC should be modularized, but that is a bigger discussion that should not be settled on the back of this PR. From a user perspective, one install that just works removes a lot of friction; not everyone knows which extra packages they need or how to manage them. The maintenance concern Bertho raises is real, but I think the answer to it is testing rather than packaging. The PR already has an offline test suite, it is just not wired into |
Fixes from the LinuxCNC#4318 review: - http_agent: escape errorCode/message in MTConnectError documents. The message embeds the raw request route, so an unescaped '<'/'&'/'"' broke well-formedness and a crafted route could forge a sibling <Error> element. - shdr_agent: sanitize '|', tab, CR and LF in SHDR values. SHDR has no escaping, so e.g. a program named 'a|b.ngc' corrupted field/line framing. - lcnc_source: read the tool pocket from stat.toolinfo(); tool_table entries carry no pocketno, so the POT Location previously reported the tool number (wrong on random toolchangers). Folds the comment lookup into the same call. - docs: register config/mtconnect.adoc and man1/mtconnect-agent.1.adoc in po4a.cfg so they are not silently dropped from translated builds. - configs: simplify the example .gitignore (drop __pycache__/*.pyc already covered upstream; ignore the whole models/ directory). - test_mtc.py: resolve config paths relative to the test file, not the CWD, so the offline suite can be run from anywhere. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
I made some minor changes to test_mtc.py to resolve config paths relative to the test file, not the CWD so the offline suite can be run from anywhere. This should make wiring into the CI easier if/when that is desired. |
Had a look at https://tracker.debian.org/pkg/three.js, so Debian uses version 111 with version 185 available. The debian/changelog lists some good guys, so I guess it is just not so easy, typically because of extra dependencies drawn in that are not in the distribution. |
|
It is not only about interfacing the softwares. Yeah, I know, we ITlers are trained to think only about technical facts. But please allow to be reminded it is also about interfacing the people behind those softwares. And I suggest to give these friendly&constructive FreeCADlers the warmest-possible welcome and to wholeheartedly embrace them. So, I would risk starting with mtconnect directly on board. From there, once everything becomes a bit more predictable, you can still transform it into a subproject or whatever the consensus may be. |
I am not sure that this is a valid concern, as the author can make changes to the code in LinuxCNC very nearly as easily as in a separate repository. Roll out to Deban can be annoyingly slow, but for useres building from source or using the LinuxCNC deb repository there probably isn't much in it. |
My thoughts exactly. Many users use master branch and build from source to follow developments in a specific GUI or feature, particularly if its an emerging one. I for one have followed master branch to get the latest version of Plasmac, QTplasmac and QTdragon on real machines. The current build process is blindingly simple thanks to input from @smoe to bring the Debian way to LinuxCNC :) |
Isn't it a bit contrary to say that we mustn't trouble the interested user with installing an external module but happily expect them to build Linuxcnc from source? |
I don't think that's what's being argued. Rod is stating the status quo: users on stable get features late and mostly care about things working; people who want shiny new things accept the risks, are more tech savvy (or learn to be), and build from source. Ease of install is only one element; adoption, visibility, long-term support, and compatibility across LinuxCNC versions all weigh in too. On version compatibility, one reminder for @sliptonic: 2.9's tool interface differs from master's (no And a note on the twin that reinforces the earlier suggestion to drop it: @smoe found Debian's libjs-three is version 111 against 185 upstream, stale precisely because packaging the JS ecosystem is a pain. So the "use the distro package" path carries its own maintenance tax, on top of the supply-chain concerns. SolidModel geometry in /probe plus the existing standard viewers (cppagent's demo twin, TrakHound, etc.) covers the use case with zero JS in tree. |
Agreed, this is the strong argument. Having the interface layer in-tree would be much easier to maintain in light of the intricate data availability and format dependencies.
That is exactly the argument why in-tree JS is both a problem and not maintainable. The JS ecosystem has a whole set of problems we do not want to get involved in or pulled into. If alternatives exist, then they could be used (maybe still out-of-tree). Otherwise, a useful (maybe packaged) procedure may be presented to users to use it out-of-tree. |
|
It's really just a matter of copying a file and a directory into his config folder and adding a few lines to the ini file. You can easily put something like that in a separate repository without much effort—and if needed, include the whole JS build stuff too. Here in the lcnc repo, everything is spread out across the directory structure, dependencies are discussed, and so on and so forth. How many users are actually going to use it anyway? BTW: I just got it running with 2.9.10; I had to comment out one line. |
|
It seems there are strong reasons not to include the twin simulation. I'm OK with that. @grandixximo I don't think a backport to 2.9 is warranted as it breaks the long standing policy of new features going in the development branch.
Not at all. Many people are running Master branch to get a specific feature before it gets to the release version. But currently I don't hear of people then installing external modules with the exception of qtpyvcp and Ethercat which we include the repositories for in our ISO.
But where?
How many people are going to use jerk limited motion? Its not about today, its about setting LinuxCNC up for the future. We live in a connected world and as more manufacturers adopt LinuxCNC, features like this will become mandatory. You've been provided a well prepared and well documented feature, why wouldn't you welcome it with open arms?
Yes its not a large extension at all and very easy to implement. |
|
How would you feel about the result of this thread not being "mtconnect-in" but "GUIs-out" and we came up with a truly modular LinuxCNC? |
Well, that is exactly the point. Once something is in-tree it sooner or later becomes the responsibility of the community.
It already works for some. Take the 'Probe Basic' GUI for example, it is a separate project and from what I see on the forum has a lot of users. I'm sure mtconnect is a great project but IMO we should start making Lcnc more modular sooner rather than later. |
what about a linuxcnc-uspace-extras Deb? Over time move stuff from the core into that Deb. I don't think any CNC controllers ship without a GUI though. But that's a much bigger project than a few lines of python proposed here. |
https://github.com/LinuxCNC/mtconnect maybe :) But that's exactly what's great about this “plugin”: Without it, you’d just see XML, and you might have to install CPPAgent, compile it, set up a database, install Grafana, configure it, set up some adapters, and so on... Users—like me—just want to see something without spending five hours installing stuff first. That’s why I liked this MTConnect implementation right away— and we talk about a file thats 12K small (twin.html), the treejs stuff is loaded from extern |
a repo here would be OK too. Perhaps there is a seperate script to install the JS stuff |
|
Let me add a thought because the implications argue for an in-tree feature. If MTConnect support becomes an out-of-the-box feature, it's use will affect what people expect from other parts of the system as well. @rod-au already mentioned getting torque data because of its usefulness in anticipating failure. That's a common use-case if users want a maintenance oriented MTConnect dashboard. Another example is tool usage time. I think this is something that linuxcnc doesn't track at all. Haas and other commercial controls do because it's such an important number for knowing when to replace tooling. The /assets route basically expose the tool table to other consumers (like CAM!!). But tool wear is notably lacking. it would be wonderful if the data included the total hours for each tool. TL;DR If MTConnect support is core, it will provide a positive signal for developing additional features that tend toward integration. That's a good thing. |
8 participants
Add native MTConnect support
A userspace, non-realtime HAL component (mtconnect-agent) that exposes LinuxCNC machine status, a full kinematic description, and tool data over the MTConnect standard — no external cppagent required. Configured from a small [MTCONNECT] INI section; the device model is generated automatically from [TRAJ]/[KINS]/[AXIS_]/[JOINT_].
Packaged as a userspace comp (bin/mtconnect-agent, lib/python/mtc, share/linuxcnc/mtconnect) with a man page, an integrator chapter, and an offline test suite in the demo config (configs/sim/axis/mtconnect).
Three commits: native agent, documentation, optional Home Assistant contrib (kept out of core since HA discovery is HA-specific). MQTT needs python3-paho-mqtt (added as a Recommends).