Connecting multiple duets to one SBC

chrishamm

@benecito Yes, it is possible in theory, but I have never tried it in practice before. You can have two instances of DuetControlServer/DuetWebServer running on the same system, but then of course with different configs for each of them. For the second Duet you will need to configure a different SPI port or at least a different CS line and a different TfrRdy GPIO pin. In addition, the second DCS instance needs to use a different base path (virtual directory of the first SD card used by RRF), a different UNIX IPC socket directory, and the second DWS instance has to be configured to use it then. See here for further details about the SPI link.

The UNIX IPC socket path directory is configured by property SocketDirectory and the virtual SD directory base path is set by BaseDirectory in /opt/dsf/conf/config.json. The same UNIX socket path has to be set using the --SocketPath <SocketDirectory>/dcs.sock argument when launching the second DWS instance as well as a different URL to listen to using the --urls http://*:8080 argument.

DCS can be started using the -c <configPath> argument to specify a different config file, so maybe just copy /opt/dsf/conf/config.json to /opt/dsf/conf/config2.json and make a second service instance to use that.

benecito

@chrishamm Thanks for the quick reply! Would this concept (in theory) be expandable to as many instances as needed until the pi runs out of power at some point?

Will try it and report back (but might take a bit)

chrishamm

@benecito Each DSF instance requires some CPU + memory to keep the OM updated and sharing the same SPI port may introduce extra lags, so I'd only connect independent SPI ports if possible. Please let me know how it goes!

benecito

@chrishamm Will report back after some experimenting, or when running into some issues

benecito

@chrishamm
I just figured out the wiring but can't start a second DCS instance.

I duplicated /opt/dsf/ in /opt/dsf2/ to keep things separate but also tried the config2.json. As neither of them worked I also tried to start the first (regular) DCS instance with

sudo systemctl start duetcontrolserver -c /opt/dsf/conf/config.json

but it returns:

systemctl: invalid option -- 'c'

Are you sure this works, or do I need to duplicate something else as well?

I managed to start one instance on spi4, but figured spi1 und 4 don't work together - will need to search for an alternate wiring scheme I guess...

benecito

@chrishamm
Managed to rewire to spi3 and have spi0 and spi3 running at the same time now. So just need to figure out how to start the second DCS

chrishamm

@benecito You need to duplicate the service file and edit it (/usr/lib/systemd/system/duetcontrolserver.service) to start with a different config. Same applies to the duetwebserver service.

benecito

@chrishamm all right - duplicated both of them. Do I need to edit anything but line 6?

[Unit]
Description=Duet Control Server
StartLimitIntervalSec=0

[Service]
ExecStart=/opt/dsf2/bin/DuetControlServer
TimeoutStopSec=15
Restart=always
Type=notify
User=dsf
Group=dsf
UMask=0002
CapabilityBoundingSet=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_SYS_TIME
AmbientCapabilities=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_SYS_TIME

[Install]
WantedBy=sysinit.target

chrishamm

@benecito It won't work that way, because DCS defaults to /opt/dsf/conf/config.json. Use

ExecStart=/opt/dsf/bin/DuetControlServer -c /opt/dsf2/conf/config.json

instead. There is no need to copy the entire dsf directory but just conf and sd in it, although it should work nevertheless. You'll need to edit the second duetwebserver service in a similar way, too.

benecito

@chrishamm unfortunately it still does not work for me...
Also can't really find anything helpful in there.

pi@CR-3D-Server:~ $ sudo systemctl start duetcontrolserver2
Job for duetcontrolserver2.service failed because the control process exited with error code.
See "systemctl status duetcontrolserver2.service" and "journalctl -xe" for details.
pi@CR-3D-Server:~ $ systemctl status duetcontrolserver2.service
● duetcontrolserver2.service - Duet Control Server
     Loaded: loaded (/lib/systemd/system/duetcontrolserver2.service; disabled; vendor preset: enabled)
     Active: activating (start) since Wed 2023-05-03 19:00:23 CEST; 1s ago
   Main PID: 14140 (DuetControlServ)
      Tasks: 9 (limit: 3597)
        CPU: 907ms
     CGroup: /system.slice/duetcontrolserver2.service
             └─14140 /opt/dsf/bin/DuetControlServer -c /opt/dsf/conf/config2.json

May 03 19:00:23 CR-3D-Server systemd[1]: Starting Duet Control Server...
May 03 19:00:24 CR-3D-Server DuetControlServer[14140]: Duet Control Server v3.4.1
May 03 19:00:24 CR-3D-Server DuetControlServer[14140]: Written by Christian Hammacher for Duet3D
May 03 19:00:24 CR-3D-Server DuetControlServer[14140]: Licensed under the terms of the GNU Public License Version 3
May 03 19:00:24 CR-3D-Server DuetControlServer[14140]: [info] Settings loaded
pi@CR-3D-Server:~ $ journalctl -xe
May 03 19:00:57 CR-3D-Server systemd[1]: duetcontrolserver2.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit duetcontrolserver2.service has entered the 'failed' state with result 'exit-code'.
May 03 19:00:57 CR-3D-Server systemd[1]: Failed to start Duet Control Server.
░░ Subject: A start job for unit duetcontrolserver2.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit duetcontrolserver2.service has finished with a failure.
░░ 
░░ The job identifier is 180062 and the job result is failed.
May 03 19:00:57 CR-3D-Server systemd[1]: duetcontrolserver2.service: Scheduled restart job, restart counter is at 2629.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ Automatic restarting of the unit duetcontrolserver2.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
May 03 19:00:57 CR-3D-Server systemd[1]: Stopped Duet Control Server.
░░ Subject: A stop job for unit duetcontrolserver2.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit duetcontrolserver2.service has finished.
░░ 
░░ The job identifier is 180128 and the job result is done.
May 03 19:00:57 CR-3D-Server systemd[1]: Starting Duet Control Server...
░░ Subject: A start job for unit duetcontrolserver2.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit duetcontrolserver2.service has begun execution.
░░ 
░░ The job identifier is 180128.
lines 3564-3600/3600 (END)

this is my duetcontrolserver2.service:

[Unit]
Description=Duet Control Server
StartLimitIntervalSec=0

[Service]
ExecStart=/opt/dsf/bin/DuetControlServer -c /opt/dsf/conf/config2.json
TimeoutStopSec=15
Restart=always
Type=notify
User=dsf
Group=dsf
UMask=0002
CapabilityBoundingSet=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_SYS_TIME
AmbientCapabilities=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_SYS_TIME

[Install]
WantedBy=sysinit.target

chrishamm

@benecito That log doesn't show quite a lot. Please run journalctl -u duetcontrolserver2 -ef and post the bit that keeps repeating. It may be a good idea to share your second config file as well (config2.json).

benecito

@chrishamm Sorry for the delay - was out of office yesterday.

I figured one part out: I messed up the SPI addresses in the config file

May 03 20:35:20 CR-3D-Server systemd[1]: duetcontrolserver2.service: Main process exited, code=exited, status=74/IOERR
May 03 20:35:20 CR-3D-Server systemd[1]: duetcontrolserver2.service: Failed with result 'exit-code'.
May 03 20:35:20 CR-3D-Server systemd[1]: Failed to start Duet Control Server.
May 03 20:35:20 CR-3D-Server systemd[1]: duetcontrolserver2.service: Scheduled restart job, restart counter is at 6981.
May 03 20:35:20 CR-3D-Server systemd[1]: Stopped Duet Control Server.
May 03 20:35:20 CR-3D-Server systemd[1]: Starting Duet Control Server...
May 03 20:35:20 CR-3D-Server DuetControlServer[26915]: Duet Control Server v3.4.1
May 03 20:35:20 CR-3D-Server DuetControlServer[26915]: Written by Christian Hammacher for Duet3D
May 03 20:35:20 CR-3D-Server DuetControlServer[26915]: Licensed under the terms of the GNU Public License Version 3
May 03 20:35:21 CR-3D-Server DuetControlServer[26915]: [info] Settings loaded
May 03 20:35:21 CR-3D-Server DuetControlServer[26915]: [info] Environment initialized
May 03 20:35:21 CR-3D-Server DuetControlServer[26915]: [fatal] Failed to open IO device: Error 2. Can not open SPI device file '/dev/spidev4.0'.
May 03 20:35:21 CR-3D-Server systemd[1]: duetcontrolserver2.service: Main process exited, code=exited, status=74/IOERR
May 03 20:35:21 CR-3D-Server systemd[1]: duetcontrolserver2.service: Failed with result 'exit-code'.
May 03 20:35:21 CR-3D-Server systemd[1]: Failed to start Duet Control Server.
May 03 20:35:21 CR-3D-Server systemd[1]: duetcontrolserver2.service: Consumed 1.017s CPU time.
May 03 20:35:21 CR-3D-Server systemd[1]: duetcontrolserver2.service: Scheduled restart job, restart counter is at 6982.
May 03 20:35:21 CR-3D-Server systemd[1]: Stopped Duet Control Server.
May 03 20:35:21 CR-3D-Server systemd[1]: duetcontrolserver2.service: Consumed 1.017s CPU time.
May 03 20:35:21 CR-3D-Server systemd[1]: Starting Duet Control Server...
May 03 20:35:21 CR-3D-Server DuetControlServer[26968]: Duet Control Server v3.4.1
May 03 20:35:21 CR-3D-Server DuetControlServer[26968]: Written by Christian Hammacher for Duet3D
May 03 20:35:21 CR-3D-Server DuetControlServer[26968]: Licensed under the terms of the GNU Public License Version 3
May 03 20:35:22 CR-3D-Server DuetControlServer[26968]: [info] Settings loaded
May 03 20:35:22 CR-3D-Server DuetControlServer[26968]: [info] Environment initialized
May 03 20:35:22 CR-3D-Server DuetControlServer[26968]: [fatal] Failed to open IO device: Error 2. Can not open SPI device file '/dev/spidev4.0'.

but hat SPI 3 enabled.

Figured out one or two more things but am stuck at starting DWC now

[Unit]
Description=Duet Web Server
Wants=duetcontrolserver.service
After=network.target duetcontrolserver.service

[Service]
WorkingDirectory=/opt/dsf2/sd/www
ExecStart=/opt/dsf/bin/DuetWebServer --SocketPath /run/dsf/dcs2.sock --urls http://*:8080
TimeoutStopSec=15
Type=notify
User=dsf
Group=dsf
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

and the corresponding config2.json:

{
  "AutoUpdateFirmware": true,
  "PluginSupport": true,
  "RootPluginSupport": false,
  "PluginsFilename": "/opt/dsf/conf/plugins.txt",
  "LogLevel": "info",
  "SocketDirectory": "/run/dsf",
  "SocketFile": "dcs2.sock",
  "StartErrorFile": "/run/dsf/dcs.err",
  "Backlog": 4,
  "SocketPollInterval": 2000,
  "BaseDirectory": "/opt/dsf/sd2",
  "PluginDirectory": "/opt/dsf/plugins",
  "NoTerminateOnReset": false,
  "HostUpdateInterval": 4000,
  "MaxMessageAge": 60,
  "SpiDevice": "/dev/spidev3.0",
  "SpiBufferSize": 8192,
  "SpiTransferMode": 0,
  "SpiFrequency": 8000000,
  "SpiConnectTimeout": 500,
  "SpiTransferTimeout": 500,
  "SpiConnectionTimeout": 4000,
  "MaxSpiRetries": 3,
  "GpioChipDevice": "/dev/gpiochip0",
  "TransferReadyPin": 27,
  "BufferedPrintCodes": 32,
  "BufferedMacroCodes": 16,
  "MaxBufferSpacePerChannel": 1536,
  "MaxCodeBufferSize": 256,
  "MaxMessageLength": 4096,
  "FirmwareComments": [

but also tried

[Unit]
Description=Duet Web Server
Wants=duetcontrolserver.service
After=network.target duetcontrolserver.service

[Service]
WorkingDirectory=/opt/dsf2/sd/www
ExecStart=/opt/dsf/bin/DuetWebServer -c /opt/dsf/conf/http2.json
TimeoutStopSec=15
Type=notify
User=dsf
Group=dsf
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

with

{
  "Logging": {
    "LogLevel": {
      "Default": "Information"
    }
  },
  "Kestrel": {
    "Endpoints": {
      "Http": {
        "Url": "http://*:8080"
      }
    }
  },
  "KeepAliveInterval": 30,
  "SessionTimeout": 8000,
  "ModelRetryDelay": 5000,
  "UseStaticFiles": true,
  "DefaultWebDirectory": "/opt/dsf/sd2/www",
  "MaxAge": 3600,
  "SocketPath": "/var/run/dsf/dcs2.sock",
  "WebSocketBufferSize": 8192
}

both fail with

pi@CR-3D-Server:~ $ systemctl status duetwebserver2.service
● duetwebserver2.service - Duet Web Server
     Loaded: loaded (/lib/systemd/system/duetwebserver2.service; disabled; vendor preset: enabled)
     Active: failed (Result: signal) since Fri 2023-05-05 10:29:11 CEST; 5s ago
    Process: 4310 ExecStart=/opt/dsf/bin/DuetWebServer -c /opt/dsf/conf/http2.json (code=killed, signal=ABRT)
   Main PID: 4310 (code=killed, signal=ABRT)
        CPU: 1.121s

May 05 10:29:11 CR-3D-Server DuetWebServer[4310]:    at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
May 05 10:29:11 CR-3D-Server DuetWebServer[4310]:    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
May 05 10:29:11 CR-3D-Server DuetWebServer[4310]:    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
May 05 10:29:11 CR-3D-Server DuetWebServer[4310]:    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
May 05 10:29:11 CR-3D-Server DuetWebServer[4310]:    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
May 05 10:29:11 CR-3D-Server DuetWebServer[4310]:    at DuetWebServer.Program.Main(String[] args) in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetWebServer/Program.cs:line 25
May 05 10:29:11 CR-3D-Server systemd[1]: duetwebserver2.service: Main process exited, code=killed, status=6/ABRT
May 05 10:29:11 CR-3D-Server systemd[1]: duetwebserver2.service: Failed with result 'signal'.
May 05 10:29:11 CR-3D-Server systemd[1]: Failed to start Duet Web Server.
May 05 10:29:11 CR-3D-Server systemd[1]: duetwebserver2.service: Consumed 1.121s CPU time.

chrishamm

@benecito Are you certain that /dev/spidev4.0 exists? Perhaps you need /dev/spidev3.0 instead.

It looks like the --urls parameter doesn't work as documented, I just checked this. You need to change the second service file to something like ExecStart=ASPNETCORE_URLS="http://*:8080" /opt/dsf/bin/DuetWebServer --SocketPath /run/dsf/dcs2.sock. Also, it looks like the Kestrel section in /opt/dsf/conf/http.json overrides the ASPNETCORE_URLS setting. I suggest you remove that section in that file and change duetwebserver.service (preferably after copying it to /etc/systemd/system first to avoid issues on upgrade) to ExecStart=ASPNETCORE_URLS="http://*:80" /opt/dsf/bin/DuetWebServer instead.

benecito

@chrishamm this throws

May 05 11:20:26 CR-3D-Server systemd[1]: /lib/systemd/system/duetwebserver2.service:8: Neither a valid executable name nor an absolute path: ASPNETCORE_URLS=http://*:8080
May 05 11:20:26 CR-3D-Server systemd[1]: duetwebserver2.service: Unit configuration has fatal error, unit will not be started.

chrishamm

@benecito Sorry, systemd requires a different format for environment variables:

Environment=ASPNETCORE_URLS="http://*:8080"
ExecStart=/opt/dsf/bin/DuetWebServer --SocketPath /run/dsf/dcs2.sock

benecito

@chrishamm
Hey,
finally managed to get to it again and without printers connected it is working right now.
Now just need to sort things in my head and make sure it also works fine with two machines actually printing at the same time.

You mentioned I should copy some files to be safe in case of an update here:

@chrishamm said in Connecting multiple duets to one SBC:

Also, it looks like the Kestrel section in /opt/dsf/conf/http.json overrides the ASPNETCORE_URLS setting. I suggest you remove that section in that file and change duetwebserver.service (preferably after copying it to /etc/systemd/system first to avoid issues on upgrade) to ExecStart=ASPNETCORE_URLS="http://*:80" /opt/dsf/bin/DuetWebServer instead.

Can you please point out again which files you would recommend to copy where?
Over all I created:
/etc/systemd/system/duetcontrolserver2.service
/etc/systemd/system/duetwebserver2.service
/opt/dsf/sd2/ (as a duplicate of /opt/dsf/sd/)
/opt/dsf/conf/config2 .json (as a mostly duplicate of /opt/dsf/conf/config.json)

and edited:
/opt/dsf/conf/config.json
/opt/dsf/conf/http.json
/etc/systemd/system/duetwebserver.service

I assume the ones I edited but did not duplicate should be duplicated as well?

chrishamm

@benecito That structure sounds reasonable to me. The service files come from /usr/share/systemd/system AFAIR.

benecito

@chrishamm quick update (and question)
We finally finished the test setup with actual printers connected. Unfortunately on pretty much any print one of the looses it's connection with a "Waiting for transfer ready pin" message.
Right now the wires are about 25cm long but well away from any other cables. We're trying to rearrange the setup to be able to use shorter cables but not sure if that's actually the issue.

Can it be somehow related to two SPI connections running at quite a bit of throughput on the same pi and thus some timing errors or so happening? In all cases the connection is back less than a second after.
Happy to hear any thoughts and maybe still get them running reliably!

benecito

@chrishamm Any thoughts on this?
We now tried to run the Duets over SBC but print over USB. This went well so far but haven't run any long term tests.
Therefore to me it sounds like a timing issue related to the transfer ready pin with the increased throughput of printing over SBC.
Can you confirm if the USB print does not go trough the SBC connection back and forth?

Still we would prefer to also print over SBC as we would have the whole ObjectModel available here.

chrishamm

@benecito Hard to say. Do you get any TfrRdy glitches in the M122 diagnostics? If yes, there is probably noise on the TfrRdy line causing the culprit. Try shielding the transfer cables and see if that makes a difference. If that doesn't work, consider trying out a different GPIO pin on the Pi side, perhaps even from a different gpiochip device if possible.