Temp0 on Toolboard V1.1 suddenly reading wrong values / offset

pasque

@t3p3tony I measured the resistance of R21 with a digital multimeter as explained.
It shows 29.6k Ohm.

T3P3Tony

@pasque please double check as it is a 2k2 resistor!

pasque

@t3p3tony I checked it 5 times now, R21 says 29.6k Ohm.
To compare, I also checked R22. This resistor shows 2.2k Ohm.

T3P3Tony

@pasque ok thanks, because it was working and then suddenly failed its not a case of the wrong resistor value being fitted so I suspect that there was a short period of over-voltage leading to overcurrent which partially damaged this resistor. The most likely cause is a intermittent short between your VIN power and the temperature input. Please check your wiring carefully, you may need to move the axis into a range of position to recreate the fault.

pasque

@t3p3tony thank you for your explanation!
Before I created this thread the first thing I checked was the wiring. I did not see any complications in my setup like blank wires or possible sources for shorts. In my opinion every wire is insulated well plus sometimes some extra insulation with a textile tape. Checked VIN and the heater wires twice. The wires of the heater cartridge are far away enough from the heating block. Also theres is enough space between the heater cartridge and the thermistor. There are no open wires of the thermistor to detect, as they are encapsulated and insulated with a sleeving.
Also during moves there is nearly no movement of the cables because they are quite good fixed in my Voron setup. Wires are only moving in the cable chains. So there is no position of the gantry where the wires could be touching anything. Furthermore the incident happened as the print head was in the middle of the print bed.

The printer was running now for nearly 4 months without any issue. On the day I flashed the new firmware the incident happened.
I also saw a similiar incident in the forum after a update to 3.4 beta 6:https://forum.duet3d.com/topic/25886/3-4-0beta6-after-update-heater-1-fault-shows-409-9-c
So could this be a firmware issue or is this a pure coincidence?
The toolboard was replaced in this case.

Also there was another case where the exact same resistor had a defect and the toolboard was replaced: https://forum.duet3d.com/topic/22754/toolboard-lc1-temp0-with-offset-or-wrong-reading/4

What should I do now?

I also added some pictures of my setup:

Thank you in advance!

Gixxerfast

This post is deleted!

T3P3Tony

@pasque yes, the more recent of those two posts that you referenced are what I had in mind when responding to you (the older one is a version 1.0 board with a different issue). In once case we have the board back already and confirmed that R21 was damaged by over voltage. That is why I suggested you check that. It may be worth checking for any continuity between the heater block and the temperature input pin, and between the heater and the heater block.

I cannot see a mechanism where the firmware could cause sufficient current to flow through this resistor to damage it. None the less I will raise it with @dc42 because it is odd that both you and one other user had the issue after upgrading to 3.4b6, although if course that could be an odd coincidence.

T3P3Tony

@pasque ok I have confirmed with @dc42 that there is no firmware related mechanism to cause sufficient current to flow to damage the resistor.

Your wiring does not well constrained but its definitely worth looking for a path between VIN and the temperature input pin. Through the heater block is the obvious option if not between wires.

pasque

@t3p3tony Thank you for checking that with @dc42 .
Although I am pretty sure there is no short, I will replace the heater cartridge and thermistor including their wires just to be sure.
Unfortunately, I now have a defective toolboard. I can possibly change to temp1, but is there any chance to repair the toolboard / temp0? SMD soldering is not one of my skills...

T3P3Tony

@pasque before you replace the heater cartridge and the thermistor please check between the heater block and temp0 and between the heater cartridge and heater block.

Regarding repair, where are you located?

pasque

@t3p3tony I checked a possible short with the multimeter between

heater block and heater cartridge - no short
heater block and temp0 (thermistor plugged in) / temp1 (thermistor plugged in) - no short
heater cartridge and temp0 (thermistor plugged in) / temp1 (thermistor plugged in) - no short

So a change of the heater cartridge wouldn't probably fix anything for the future.

I'm located near Stuttgart, southern Germany. Bought the Toolboard from Dold Mechatronik.

pasque

@t3p3tony Checking POWER VIN+ to VSSA of the temp1 and temp0 pins, the multimeter shows a connection / short. Is this intended?

T3P3Tony

@pasque what resistance are you seeing between those pins?

Ground to VSSA should be low resistance (there is only 10R and a fuse between them).

VSSA to VIN should be high resistance or open.

pasque

@t3p3tony sorry, I need to apologize. I think I din't probe that good. I checked it now 3 times again and I think it matches your answer now:

VSSA to VIN+ = ~390 Ohm
VSSA to GND = 10 Ohm

I also did some further testing that last hour.

I flashed 3.4 beta 6 again and reconfigured to temp1 input (that input that isn't broken right now):

1.12.2021, 18:43:39	M122 M122 B121
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.0beta6 (2021-11-06 11:39:32) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 2GMYT-T296U-D65J0-40KM4-1F03Z-7F950
Used output buffers: 3 of 40 (13 max)
=== RTOS ===
Static ram: 103876
Dynamic ram: 116860 of which 24 recycled
Never used RAM 20952, free system stack 200 words
Tasks: NETWORK(ready,12.1%,264) HEAT(notifyWait,0.0%,374) Move(notifyWait,0.0%,361) CanReceiv(notifyWait,0.0%,797) CanSender(notifyWait,0.0%,372) CanClock(delaying,0.0%,351) TMC(notifyWait,1.1%,114) MAIN(running,84.5%,428) IDLE(ready,1.4%,29) AIN(delaying,0.8%,274), total 100.0%
Owned mutexes: WiFi(NETWORK) HTTP(MAIN)
=== Platform ===
Last reset 00:00:22 ago, cause: software
Last software reset at 2021-12-01 18:43, reason: User, GCodes spinning, available RAM 20908, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 22998, completed 22997, timed out 0, errs 0
Step timer max interval 751
MCU temperature: min 27.1, current 28.2, max 28.4
Supply voltage: min 24.1, current 24.1, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Driver 0: pos 0, standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 2046, writes 11, timeouts 0, DMA errors 0
Driver 1: pos 0, standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 2045, writes 11, timeouts 0, DMA errors 0
Driver 2: pos 0, standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 2045, writes 11, timeouts 0, DMA errors 0
Driver 3: pos 0, standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 2045, writes 11, timeouts 0, DMA errors 0
Driver 4: pos 0, standstill, SG min 0, read errors 0, write errors 1, ifcnt 29, reads 2047, writes 9, timeouts 0, DMA errors 0
Driver 5: pos 0, standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 2045, writes 11, timeouts 0, DMA errors 0
Driver 6: pos 0, standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 2045, writes 11, timeouts 0, DMA errors 0
Date/time: 2021-12-01 18:43:37
Cache data hit count 42338877
Slowest loop: 4.56ms; fastest: 0.13ms
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 3.2ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1, chamber heaters -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is ready with "M122 " in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 216, received 454, lost 0, longest wait 2ms for reply type 6049, peak Tx sync delay 606, free buffers 17 (min 16), ts 115/114/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 15.36ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.26
WiFi MAC address f0:08:d1:02:e8:ee
WiFi Vcc 3.33, reset reason Power up
WiFi flash size 2097152, free heap 22864
WiFi IP address 192.168.178.88
WiFi signal strength -60dBm, mode 802.11n, reconnections 0, sleep mode modem
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.4.0beta6 (2021-11-06 11:56:11)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
Never used RAM 2576, free system stack 2788 words
Tasks: Move(notifyWait,0.0%,153) HEAT(notifyWait,0.2%,115) CanAsync(notifyWait,0.0%,64) CanRecv(notifyWait,0.0%,77) CanClock(notifyWait,0.0%,64) ACCEL(notifyWait,0.0%,61) TMC(delaying,3.0%,67) MAIN(running,91.9%,350) IDLE(ready,0.0%,27) AIN(delaying,4.9%,148), total 100.0%
Last reset 00:00:23 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 423.0 steps/mm,standstill, SG min 0, read errors 0, write errors 1, ifcnt 45, reads 11636, writes 11, timeouts 0, DMA errors 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -2/6, peak Rx sync delay 192, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 23.9, current 23.9, max 23.9
MCU temperature: min 34.2C, current 34.7C, max 34.7C
Last sensors broadcast 0x00000002 found 1 80 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 476, send timeouts 0, received 223, lost 0, free buffers 37, min 37, error reg 110000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0
Accelerometer detected: yes, status: 00
I2C bus errors 0, naks 3, other errors 0

And I started a print. After about 6mins the exact same incident happened again as on sunday: There was a fault of the sensor:

1.12.2021, 19:04:31	Connection established
1.12.2021, 19:04:28	Connection interrupted, attempting to reconnect...
1.12.2021, 19:04:19	Error: Failed to switch off remote heater 1: 
Cancelled printing file 0:/gcodes/Fab365_Bottom.gcode, print time was 0h 5m
1.12.2021, 19:04:19	Emergency stop, attemping to reconnect...
1.12.2021, 18:58:46	Leadscrew adjustments made: 0.063 -0.012 -0.026 0.071, points used 4, (mean, deviation) before (0.035, 0.030) after (0.000, 0.000)
1.12.2021, 18:58:22	File 0:/gcodes/Fab365_Bottom.gcode selected for printing

This time, the temp1 input seems to be okay, resistance of temp1 is normal, temperature was as intended to be. Didn't change anything in the hardware configuration.
I flashed back to 3.3 again. Started the exact same file again.

1.12.2021, 19:26:12	Leadscrew adjustments made: -0.277 -0.254 -0.327 1.087, points used 4, (mean, deviation) before (0.146, 0.401) after (0.000, 0.000)
1.12.2021, 19:25:49	M32 "0:/gcodes/Fab365_Bottom.gcode"
File 0:/gcodes/Fab365_Bottom.gcode selected for printing

1.12.2021, 19:29:17	M122 M122 B121
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.3 (2021-06-15 21:46:11) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: 2GMYT-T296U-D65J0-40KM4-1F03Z-7F950
Used output buffers: 3 of 40 (12 max)
=== RTOS ===
Static ram: 102724
Dynamic ram: 113112 of which 144 recycled
Never used RAM 24868, free system stack 132 words
Tasks: NETWORK(ready,15.4%,240) HEAT(delaying,0.0%,344) Move(notifyWait,0.1%,272) CanReceiv(notifyWait,0.0%,773) CanSender(notifyWait,0.0%,363) CanClock(delaying,0.0%,347) TMC(notifyWait,1.2%,106) MAIN(running,82.3%,426) IDLE(ready,0.1%,29) AIN(delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK) HTTP(MAIN)
=== Platform ===
Last reset 00:09:54 ago, cause: software
Last software reset at 2021-12-01 19:19, reason: User, GCodes spinning, available RAM 28116, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 594986, completed 594985, timed out 0, errs 0
Step timer max interval 870
MCU temperature: min 28.4, current 32.8, max 33.0
Supply voltage: min 24.0, current 24.1, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/4/4, gc cycles 0
Driver 0: position 55754, standstill, SG min/max 0/508, read errors 0, write errors 1, ifcnt 39, reads 54050, writes 15, timeouts 0, DMA errors 0
Driver 1: position 7320, standstill, SG min/max 0/492, read errors 0, write errors 1, ifcnt 39, reads 54050, writes 15, timeouts 0, DMA errors 0
Driver 2: position 80, standstill, SG min/max 0/482, read errors 0, write errors 1, ifcnt 39, reads 54050, writes 15, timeouts 0, DMA errors 0
Driver 3: position 0, standstill, SG min/max 0/474, read errors 0, write errors 1, ifcnt 39, reads 54050, writes 15, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 29, reads 54056, writes 9, timeouts 0, DMA errors 0
Driver 5: position 0, ok, SG min/max 0/492, read errors 0, write errors 1, ifcnt 39, reads 54050, writes 15, timeouts 0, DMA errors 0
Driver 6: position 0, ok, SG min/max 0/432, read errors 0, write errors 1, ifcnt 39, reads 54050, writes 15, timeouts 0, DMA errors 0
Date/time: 2021-12-01 19:29:15
Cache data hit count 1065433227
Slowest loop: 215.67ms; fastest: 0.12ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 3.9ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 215333ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 2620, completed moves 2580, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 1], CDDA state 3
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 1
Movement lock held by null
HTTP is ready with "M122 " in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X196.459 Y167.695 E0.02106" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 7888, received 7168, lost 0, longest wait 2ms for reply type 6049, peak Tx sync delay 271, free buffers 17 (min 13), ts 2975/2974/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 208.88ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.26
WiFi MAC address f0:08:d1:02:e8:ee
WiFi Vcc 3.33, reset reason Power up
WiFi flash size 2097152, free heap 26440
WiFi IP address 192.168.178.88
WiFi signal strength -58dBm, mode 802.11n, reconnections 0, sleep mode modem
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0

Diagnostics for board 121:
Duet TOOL1LC firmware version 3.3 (2021-06-15 16:12:58)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
Never used RAM 2624, free system stack 2745 words
Tasks: Move(notifyWait,0.2%,99) HEAT(delaying,0.2%,109) CanAsync(notifyWait,0.0%,61) CanRecv(notifyWait,0.0%,74) CanClock(notifyWait,0.0%,65) ACCEL(notifyWait,0.0%,61) TMC(delaying,2.8%,57) MAIN(running,91.8%,352) IDLE(ready,0.0%,27) AIN(delaying,5.0%,142), total 100.0%
Last reset 00:09:55 ago, cause: software
Last software reset data not available
Driver 0: position 94168, 423.0 steps/mm, ok, SG min/max 0/16, read errors 0, write errors 1, ifcnt 48, reads 35274, writes 14, timeouts 0, DMA errors 0, steps req 109058 done 109015
Moves scheduled 2487, completed 2485, in progress 1, hiccups 0, step errors 0, maxPrep 572, maxOverdue 3, maxInc 2, mcErrs 0, gcmErrs 0
Peak sync jitter -4/9, peak Rx sync delay 209, resyncs 0/0, next step interrupt due in 400 ticks, enabled
VIN: 23.7V
MCU temperature: min 40.6C, current 44.7C, max 44.7C
Ticks since heat task active 196, ADC conversions started 594830, completed 594829, timed out 0, errs 0
Last sensors broadcast 0x00000002 found 1 202 ticks ago, loop time 0
CAN messages queued 7191, send timeouts 0, received 7895, lost 0, free buffers 37, min 36, error reg 110000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 529, adv 22117/74617
Accelerometer detected: yes, status: 00
I2C bus errors 0, naks 0, other errors 0

The printer printed the file without any error, no drops in temperature or any unusual behavoiur.
I'm really confused right now

Edit: with 3.4 beta 6 Input Shaping was activated, in case that matters:

M593 P"MZV" F61

T3P3Tony

@pasque this really has all the symptoms of an intermittent fault - I definitely don't think this is a firmware issue (even though it coincided with a firmware upgrade initially).

@pasque said in Temp0 on Toolboard V1.1 suddenly reading wrong values / offset:

1.12.2021, 19:04:19 Error: Failed to switch off remote heater 1:

This is interesting as well - did you notice any CAN communication drops? I wonder if the fault caused the tool board processor to shut down temporarily.

pasque

@t3p3tony I just printed another file with 3.3 without any issue for over 3h.

To investigate I will flash 3.4 beta 6 tomorrow again to see if this may happen again.
I did not notice any CAN communication issues - is there a way to detect them?

T3P3Tony

@pasque issues with CAN communication are reported in the M122 of the mainboard and of the toolboard.

pasque

Hi @t3p3tony, I disassembled the print head and checked every wiring. Therey may have been a loose wire behind the tool fan, I'm not completely sure. I also changed the wires of the thermistor as I suspected these most likely to be the problem.
Since then the printer worked without any further incident - I will carefully check the next prints.
I can't really comprehend why the resistor R21 of temp0 is damaged now but I would really like to have it repaired because the temp0 port is at a better position for my wiring. Any suggestions?