Solved!

Found that a set screw had backed out that was holding the fixed pulley in place on the stepper shaft.

We found it odd In the way it was manifesting.

Thank you everyone for the assistance.

Thank you, I'm running DCS in debug mode - and will be running jobs beginning tomorrow morning with the last jobs to see if I can reproduce the issue.

Syslog with the observed issue https://drive.google.com/file/d/1qGrD7uS_jXZGs1kyyUrE6AGbiEL8Fvxc/view?usp=sharing

Great, that's from the kernel log - I'm pulling the syslog to see if there's anything there.

JubileePiLogDump.txt

@Phaedrux Thanks for the info - I'm running the debug monitoring now - Found what looks like an out of memory dump in the Pi's logs at or about the time of the last failure - Should I post it here?
Thanks!

@Wally Thanks for the suggestion but I was really hoping for some diagnostics / troubleshooting. Does anyone know if we can connect the Pi to the Duet3 via USB while in operation to capture the debug output on the USB port? I've already set the DSF / DWC to debug logging and will be trying some increasingly longer prints to see if I can reproduce the issue.

@Wally Pi is powered by the Duet3 per the documentation.

Page reloads did not work.
Browser restart did not work.
Browser cache clear and restart did not work.

Power cycle of entire system required to regain access to DWC as stated in initial post.

No errors recorded in eventlog.txt

No errors recorded in console.

DWC hung again - required reboot of pi to restore access

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.1.1 running on Duet 3 MB6HC v0.6 or 1.0 (SBC mode)
Board ID: 08DJM-956L2-G43S4-6J9DL-3SJ6S-1866H
Used output buffers: 1 of 40 (14 max)
=== RTOS ===
Static ram: 154604
Dynamic ram: 163732 of which 20 recycled
Exception stack ram used: 520
Never used ram: 74340
Tasks: NETWORK(ready,1972) HEAT(blocked,1088) CanReceiv(suspended,3420) CanSender(suspended,1428) CanClock(blocked,1436) TMC(blocked,68) MAIN(running,4936) IDLE(ready,76)
Owned mutexes:
=== Platform ===
Last reset 09:06:56 ago, cause: power up
Last software reset at 2020-07-22 19:13, reason: User, spinning module LinuxInterface, available RAM 74300 bytes (slot 2)
Software reset code 0x0010 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0444a000 BFAR 0x00000000 SP 0xffffffff Task MAIN
Error status: 0
MCU temperature: min 18.1, current 39.6, max 41.1
Supply voltage: min 23.9, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 11.9, current 12.0, max 12.1, under voltage events: 0
Driver 0: standstill, reads 26677, writes 27 timeouts 0, SG min/max 0/1023
Driver 1: standstill, reads 26681, writes 23 timeouts 0, SG min/max 0/55
Driver 2: standstill, reads 26678, writes 27 timeouts 0, SG min/max 0/1023
Driver 3: standstill, reads 26678, writes 27 timeouts 0, SG min/max 0/1023
Driver 4: standstill, reads 26679, writes 27 timeouts 0, SG min/max 0/1023
Driver 5: standstill, reads 26679, writes 27 timeouts 0, SG min/max 0/1023
Date/time: 2020-07-25 19:05:43
Slowest loop: 10.81ms; fastest: 0.14ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 375, MinFreeDm: 322, MaxWait: 3079109ms
Bed compensation in use: mesh, comp offset 0.008
=== MainDDARing ===
Scheduled moves: 45996, completed moves: 45996, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0 CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is ready with "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon* is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 2.21ms; fastest: 0.01ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions
HTTP sessions: 0 of 8

• Ethernet -
  State: disabled
  Error counts: 0 0 0 0 0
  Socket states: 0 0 0 0 0 0 0 0
  === CAN ===
  Messages sent 175312, longest wait 2ms for type 6011
  === Linux interface ===
  State: 0, failed transfers: 0
  Last transfer: 17ms ago
  RX/TX seq numbers: 6486/19231
  SPI underruns 0, overruns 0
  Number of disconnects: 8
  Buffer RX/TX: 0/0-0
  === Duet Control Server ===
  Duet Control Server v3.1.1
  Code buffer space: 4096
  Configured SPI speed: 8000000 Hz
  Full transfers per second: 29.63

config.g

I've run into this again today where the machine stopped mid-print and DWC became unresponsive. Heaters and fans remained on. No errors generated in the console or in DWC. Was able to ssh into pi, was able to VNC into pi, DWC completely unresponsive. CAN-FD sync light blinking in time, no error lights on Duet3.

Required power cycle to regain access to DWC - print was lost.

How do I troubleshoot this issue - the console log does not appear to be functioning properly as there are no entries.
Can I connect the Pi to the Duet USB and set up debugging and capture the USB output on the PI?
I currently do not trust the setup for unattended operation.

system configuration:
Duet3 6HC (SBC Pi4 4GB) + 3HC + 4 x Tool boards (with distro board)
RRF 3.1.1 + DuetPi with updates
CoreXY
XYZE is x16 microstepping Interpolated
U is x4 microstepping interpolated

I'd like to add that I am experiencing the same issue - Currently running current code on a Duet3 6HC (SBC Pi4 4GB) + 3HC + 4 x tool boards with distro board. Upon powering on the system the eventlog.txt records:

2020-07-24 16:21:59 Event logging started
2020-07-24 16:21:59 Event logging stopped

Here I issued a manual M929 S1

2020-07-24 19:21:31 Event logging started
2020-07-24 19:21:31 Event logging stopped