SD card gets full after printing through DWC

Ping

Hello,

I have encountered some problems regarding SD card.
I use 6HC board with SBC.

After some printing through Duet web control I always end up with full SD card and I'm not able to find out where to delete cache or any other leftover files. How should I solve this situation? Last time I totally reformatted SD card as I thought there may be more problems related to that but unfortunately all the reinstallation work didn't help and now I have the same problem on my other printer.

I have tried "sudo apt-get clean", then open the SD card manually but I couldn't find where the storage is full.

DF -h command provides me with this information:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G   14G     0 100% /
devtmpfs        430M     0  430M   0% /dev
tmpfs           462M   16M  447M   4% /dev/shm
tmpfs           462M  6.4M  456M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1  247M   60M  187M  25% /boot
tmpfs            93M  8.0K   93M   1% /run/user/1000

I am not able to reach /dev/root folder

M122 diagnostic:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.4.0 (2022-03-15 18:57:24) running on Duet 3 MB6HC v1.01 or later (SBC mode)
Board ID: 08DJM-9P63L-DJMSS-6J9F0-3SD6K-9UDB9
Used output buffers: 1 of 40 (26 max)
=== RTOS ===
Static ram: 151000
Dynamic ram: 67132 of which 16 recycled
Never used RAM 129284, free system stack 132 words
Tasks: SBC(resourceWait:,0.7%,436) HEAT(notifyWait,0.1%,321) Move(notifyWait,0.5%,237) CanReceiv(notifyWait,0.0%,772) CanSender(notifyWait,0.0%,348) CanClock(delaying,0.0%,339) TMC(notifyWait,8.0%,92) MAIN(running,90.7%,923) IDLE(ready,0.0%,30), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 03:44:58 ago, cause: power up
Last software reset at 2022-04-06 00:34, reason: User, GCodes spinning, available RAM 129380, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
Step timer max interval 973
MCU temperature: min 25.7, current 26.2, max 26.6
Supply voltage: min 23.3, current 23.6, max 23.8, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 1.0, current 12.1, max 12.5, under voltage events: 1
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/114/114, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, mspos 8, reads 54347, writes 11 timeouts 0
Driver 1: standstill, SG min 0, mspos 8, reads 54347, writes 11 timeouts 0
Driver 2: standstill, SG min 0, mspos 8, reads 54346, writes 11 timeouts 0
Driver 3: standstill, SG min 0, mspos 8, reads 54346, writes 11 timeouts 0
Driver 4: standstill, SG min 0, mspos 8, reads 54346, writes 11 timeouts 0
Driver 5: standstill, SG min 0, mspos 8, reads 54346, writes 11 timeouts 0
Date/time: 2022-04-11 04:34:46
Slowest loop: 25.88ms; fastest: 0.04ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 16, maxWait 363050ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 7758, completed 7758, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 13], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 4 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 67742, received 210444, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 282, free buffers 50 (min 35), ts 21921/21921/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 4, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 6356/6356
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0, IAP RAM available 0x2b880
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server v3.4.0
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 2
Full transfers per second: 39.35, max time between full transfers: 87.9ms, max pin wait times: 70.0ms/17.6ms
Codes per second: 1.34
Maximum length of RX/TX data transfers: 6504/1664

M39 sd card diagnostic:

M39
SD card in slot 0: capacity 15.00Gb, partition size 15.00Gb,free space 0.00Gb, speed MBytes/sec

I was unable to find clear instructions on how and where all the files/cache is stored and how to reach that and clean.

In general DWC settings I have options for storing settings and files in local storage. How there settings behave when turned on?

Thank you very much

Phaedrux

Do you have logging enabled?

Do you have a larger SD card to use?

Either way, I would suggest trying a fresh download of DuetPi and re-imaging the card after backing up your config. See how much space is left available on a fresh install. How large are the gcode files you are printing? There may not be much space left over with 16gb to start with.

chrishamm

@ping It sounds very much like you still have the DCS log level set to debug (check logLevel in /opt/dsf/conf/config.json) and that rsyslog is still running on your image. I strongly suggest disabling it (sudo systemctl stop rsyslog && sudo systemctl disable rsyslog) and cleaning up the logs (sudo rm /var/log/*.log). That should free up your SD card again.

Ping

@Phaedrux I have checked if I have logging enabled by @chrishamm method but found out that it is not enabled:

 GNU nano 3.2               /opt/dsf/conf/config.json

{
  "AutoUpdateFirmware": true,
  "PluginSupport": true,
  "RootPluginSupport": false,
  "PluginsFilename": "/opt/dsf/conf/plugins.txt",
  "LogLevel": "info",
  "SocketDirectory": "/run/dsf",
  "SocketFile": "dcs.sock",
  "StartErrorFile": "/run/dsf/dcs.err",
  "Backlog": 4,
  "SocketPollInterval": 2000,
  "BaseDirectory": "/opt/dsf/sd",
  "PluginDirectory": "/opt/dsf/plugins",
  "NoTerminateOnReset": false,
  "HostUpdateInterval": 4000,
  "MaxMessageAge": 60,
  "SpiDevice": "/dev/spidev0.0",
  "SpiBufferSize": 8192,
  "SpiTransferMode": 0,

Gcode files for printing are usually not so big ranging from 2 to 10Mb.

To be honest I wouldn't want to re-image the card as it may lead to more problem solving the same way like when I reformatted the card for another printer. I have used 32 Gb card for another printer.

Fresh install and maximum one week of use gives usage of sd card like this:

pi@~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        28G  6.6G   20G  26% /
devtmpfs        430M     0  430M   0% /dev
tmpfs           462M   25M  437M   6% /dev/shm
tmpfs           462M  6.4M  456M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1  2.0G   60M  2.0G   3% /boot
tmpfs            93M  4.0K   93M   1% /run/user/1000

I have tried to stop rsyslog and clean up the logs with no success:

pi@:~ $ sudo systemctl stop rsyslog
Warning: Stopping rsyslog.service, but it can still be activated by:
  syslog.socket
pi@:~ $ sudo systemctl disable rsyslog
Synchronizing state of rsyslog.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable rsyslog
Removed /etc/systemd/system/syslog.service.
Removed /etc/systemd/system/multi-user.target.wants/rsyslog.service.
pi@:~ $ sudo rm /var/log/*.log
pi@:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G   14G     0 100% /
devtmpfs        430M     0  430M   0% /dev
tmpfs           462M   16M  446M   4% /dev/shm
tmpfs           462M   13M  450M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1  247M   60M  187M  25% /boot
tmpfs            93M  8.0K   93M   1% /run/user/1000

I believe that is related to web control cache as I upload files through web control and then delete them after use and it still keeps SD card full.

Looking forward to any new suggestions,

Thank you very much for fast replies guys:)

chrishamm

@ping You can check the size of directories using the du command. What totals do you get for

du -hc /var/log
du -hc /opt/dsf/sd

Ping

@chrishamm
du -hc /var/log
7.6M total

du -hc /opt/dsf/sd
34M total

chrishamm

@ping Ok, so neither the logs nor the printer files take up that much space. You can run the du -hc command again on e.g. /home, /var, /opt, /usr, and check if any of those are excessively large.

Ping

@chrishamm

/home approx 200Mb
/var 11Gb
/opt approx 200Mb
/usr 2.8 Gb

pi@:~ $ du -hc /var
4.0K    /var/lib/hp
4.0K    /var/lib/apt/periodic
du: cannot read directory '/var/lib/apt/lists/partial': Permission denied
4.0K    /var/lib/apt/lists/partial
4.0K    /var/lib/apt/lists/auxfiles
69M     /var/lib/apt/lists
4.0K    /var/lib/apt/mirrors/partial
8.0K    /var/lib/apt/mirrors
69M     /var/lib/apt
4.0K    /var/lib/dhcp
604K    /var/lib/usbutils
4.0K    /var/lib/dbus
4.0K    /var/lib/snmp
8.0K    /var/lib/blueman
4.0K    /var/lib/colord/icc
du: cannot read directory '/var/lib/colord/.cache': Permission denied
4.0K    /var/lib/colord/.cache
56K     /var/lib/colord
8.0K    /var/lib/ispell
4.0K    /var/lib/dpkg/parts
320K    /var/lib/dpkg/alternatives
48K     /var/lib/dpkg/triggers
4.0K    /var/lib/dpkg/updates
47M     /var/lib/dpkg/info
50M     /var/lib/dpkg
4.0K    /var/lib/sudo/lectured
8.0K    /var/lib/sudo
du: cannot read directory '/var/lib/lightdm': Permission denied
4.0K    /var/lib/lightdm
28K     /var/lib/pam
4.0K    /var/lib/openbox
4.0K    /var/lib/git
24K     /var/lib/ucf/cache
60K     /var/lib/ucf
du: cannot read directory '/var/lib/bluetooth': Permission denied
4.0K    /var/lib/bluetooth
4.0K    /var/lib/python
4.0K    /var/lib/ntp
4.0K    /var/lib/nfs/sm.bak
4.0K    /var/lib/nfs/sm
12K     /var/lib/nfs
du: cannot read directory '/var/lib/polkit-1': Permission denied
4.0K    /var/lib/polkit-1
8.0K    /var/lib/alsa
12K     /var/lib/dictionaries-common/wordlist
8.0K    /var/lib/dictionaries-common/aspell
24K     /var/lib/dictionaries-common
4.0K    /var/lib/initramfs-tools
16K     /var/lib/dhcpcd5
36K     /var/lib/PackageKit
8.0K    /var/lib/plymouth
du: cannot read directory '/var/lib/private': Permission denied
4.0K    /var/lib/private
12K     /var/lib/sgml-base
4.0K    /var/lib/vim/addons
8.0K    /var/lib/vim
8.0K    /var/lib/xkb
4.0K    /var/lib/emacsen-common/state/package/installed
8.0K    /var/lib/emacsen-common/state/package
4.0K    /var/lib/emacsen-common/state/flavor/installed
8.0K    /var/lib/emacsen-common/state/flavor
20K     /var/lib/emacsen-common/state
24K     /var/lib/emacsen-common
du: cannot read directory '/var/lib/udisks2': Permission denied
4.0K    /var/lib/udisks2
4.0K    /var/lib/man-db
4.0K    /var/lib/systemd/timesync
136K    /var/lib/systemd/catalog
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/halt.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/poweroff.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/graphical.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/network-online.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/bluetooth.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/remote-fs.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/sockets.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/reboot.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/sysinit.target.wants
4.0K    /var/lib/systemd/deb-systemd-helper-enabled/printer.target.wants
236K    /var/lib/systemd/deb-systemd-helper-enabled
4.0K    /var/lib/systemd/coredump
16K     /var/lib/systemd/rfkill
4.0K    /var/lib/systemd/timers
408K    /var/lib/systemd
4.0K    /var/lib/usb_modeswitch
4.1M    /var/lib/aspell
16K     /var/lib/xml-core
12K     /var/lib/ghostscript/fonts
20K     /var/lib/ghostscript/CMap
36K     /var/lib/ghostscript
8.0K    /var/lib/logrotate
4.0K    /var/lib/raspberrypi/bootloader/backup
8.0K    /var/lib/raspberrypi/bootloader
12K     /var/lib/raspberrypi
4.0K    /var/lib/misc
11G     /var/lib/motion
11G     /var/lib
3.7M    /var/backups
du: cannot read directory '/var/cache/apt/archives/partial': Permission denied
4.0K    /var/cache/apt/archives/partial
36K     /var/cache/apt/archives
40K     /var/cache/apt
du: cannot read directory '/var/cache/apparmor/ea9ed67a.0': Permission denied
4.0K    /var/cache/apparmor/ea9ed67a.0
8.0K    /var/cache/apparmor
du: cannot read directory '/var/cache/cups': Permission denied
4.0K    /var/cache/cups
8.0K    /var/cache/unbound
du: cannot read directory '/var/cache/lightdm': Permission denied
4.0K    /var/cache/lightdm
4.4M    /var/cache/debconf
32K     /var/cache/man/ru
52K     /var/cache/man/de
24K     /var/cache/man/sr
24K     /var/cache/man/sl
24K     /var/cache/man/da
24K     /var/cache/man/fr.ISO8859-1
24K     /var/cache/man/id
28K     /var/cache/man/sv
24K     /var/cache/man/cs
32K     /var/cache/man/nl
28K     /var/cache/man/es
24K     /var/cache/man/fi
24K     /var/cache/man/pt_BR
52K     /var/cache/man/fr
28K     /var/cache/man/pl
24K     /var/cache/man/zh_TW
24K     /var/cache/man/tr
24K     /var/cache/man/zh_CN
28K     /var/cache/man/ja
24K     /var/cache/man/fr.UTF-8
24K     /var/cache/man/hu
28K     /var/cache/man/it
24K     /var/cache/man/ko
28K     /var/cache/man/pt
1.6M    /var/cache/man
40K     /var/cache/dictionaries-common
4.0K    /var/cache/PackageKit/downloads
8.0K    /var/cache/PackageKit
du: cannot read directory '/var/cache/private': Permission denied
4.0K    /var/cache/private
du: cannot read directory '/var/cache/ldconfig': Permission denied
4.0K    /var/cache/ldconfig
240K    /var/cache/fontconfig
6.2M    /var/cache
4.0K    /var/local
du: cannot read directory '/var/spool/rsyslog': Permission denied
4.0K    /var/spool/rsyslog
du: cannot read directory '/var/spool/cups': Permission denied
4.0K    /var/spool/cups
du: cannot read directory '/var/spool/cron/crontabs': Permission denied
4.0K    /var/spool/cron/crontabs
8.0K    /var/spool/cron
20K     /var/spool
4.0K    /var/opt
4.0K    /var/mail
du: cannot read directory '/var/tmp/systemd-private-7d85e4b769d14331b6be8feb25ac6aa7-systemd-timesyncd.service-iBMXya': Permission denied
4.0K    /var/tmp/systemd-private-7d85e4b769d14331b6be8feb25ac6aa7-systemd-timesyncd.service-iBMXya
du: cannot read directory '/var/tmp/systemd-private-7d85e4b769d14331b6be8feb25ac6aa7-colord.service-8XTOUG': Permission denied
4.0K    /var/tmp/systemd-private-7d85e4b769d14331b6be8feb25ac6aa7-colord.service-8XTOUG
du: cannot read directory '/var/tmp/systemd-private-7d85e4b769d14331b6be8feb25ac6aa7-rtkit-daemon.service-U55l0X': Permission denied
4.0K    /var/tmp/systemd-private-7d85e4b769d14331b6be8feb25ac6aa7-rtkit-daemon.service-U55l0X
16K     /var/tmp
4.0K    /var/log/hp/tmp
8.0K    /var/log/hp
116K    /var/log/apt
72K     /var/log/cups
du: cannot read directory '/var/log/lightdm': Permission denied
4.0K    /var/log/lightdm
du: cannot read directory '/var/log/private': Permission denied
4.0K    /var/log/private
4.0K    /var/log/proftpd
1.6M    /var/log/motion
7.6M    /var/log
11G     /var
11G     total

It seems like 11G is used by hidden files or do I understand something wrong?

chrishamm

@ping Not really - it seems you have a webcam set up on your system and it keeps writing to /var/lib/motion:

11G     /var/lib/motion

That's far too much. I suggest you uninstall it and clean up the motion directory once, then start over and reconfigure motion so that it doesn't write photos automatically:

sudo apt purge motion
sudo rm -rf /var/lib/motion

Ping

@chrishamm
I remember configuring motion to not store photos but it still stores something. I will investigate it more.

SD card is now back to normal:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  3.5G   11G  26% /
devtmpfs        430M     0  430M   0% /dev
tmpfs           462M   18M  444M   4% /dev/shm
tmpfs           462M   13M  450M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1  247M   60M  187M  25% /boot
tmpfs            93M  8.0K   93M   1% /run/user/1000

Thank you very much for solved issue!