DuetLapse3

[GoremanX]

@stuartofmt One observation: isn't it possible to pull the file name of the current print job through the object model? If so, instead of just processid, could the directory be named "jobfilename-processid"? That way, the directory is easily identifiable as the result of a specific job, but there's still a unique number appended to it for when the same job is run more than once

[stuartofmt]

@GoremanX said in DuetLapse3:

@stuartofmt One observation: isn't it possible to pull the file name of the current print job through the object model? If so, instead of just processid, could the directory be named "jobfilename-processid"? That way, the directory is easily identifiable as the result of a specific job, but there's still a unique number appended to it for when the same job is run more than once

I do not know what you mean by the jobfilename? What defines a job is the options and that would be incredibly long as a directory name. I'm pretty sure you are not proposing directory paths that look like this: ????.

/home/stuart/Lapse/
                  Stuart.localdomain/
                                    192-168-86-235/
                                                 -duet 192.168.86.235 -instances many -dontwait -port 8083 -standby -seconds 15 -detect none -camera1 stream -weburl1 http://192.168.86.230:8081/stream.mjpg

Note that the http listener provides info like this:

[GoremanX]

@stuartofmt Isn't this accessible through the API, same way the current machine state is?

[stuartofmt]

@GoremanX said in DuetLapse3:

@stuartofmt Isn't this accessible through the API, same way the current machine state is?

Ah - I see now what you mean by jobfilename. It's easy enough to get and I see how it would aid in reading the directories. There is unfortunately an issue of timing.

That the information is only available after the print job has started - so it's not guaranteed to be available when DuetLapse3 starts. Also - even when the print job does start - there are some cases (e.g. -dontwait - seconds nnn) when you could not reasonably create the directory after the job starts.

Having said that - I'll add it to the directory name (processid-jobname) if available at the time the directory is created and delay the creation of the directory until the last possible moment. i.e. immediately before the first image is captured).

Sound like a plan ?

[stuartofmt]

I am pleased to release Version 3.4.0. This was more difficult than I first thought. The restructuring of the directories (in particular the cleanup) introduced the need for some tight internal logic that was tricky to test.
A couple of the additional (and seemingly simple) changes ended up being quite a challenge to figure out.
The core functions were not changes so this release should be solid in that regard. Thanks to those that provided feedback and ideas on the interim design.

Enjoy.

###Version 3.4.0###
[1] Completely revised the directory and file-naming structure to facilitate many-to-many relationships between computers running DuetLapse3, Duet Printers and multiple instances of DuetLapse3.
[2] Changed how http terminate requests were handled for better cross platform compatibility.
[3] Added the ability to navigate the directory structure from a browser (new button). Also in startDuetLapse3.
[4] Added three new options: -keeplogs, -deletepics and -novideo
[5] Made some cosmetic changes to the html pages.

The new directory structure is (with repeating units [] as appropriate to your use-case)

basedir/
   |
   computername/
   |   |
   |   duetip/
   |      |
   |      |processid_jobname/         #_jobname if available
   |      |
   |      |[/processid_jobname}
   |
   |
   [computername 
     #repeat of structure above
   ]         

For many (most ?) users it will simply look like this

basedir/
   |
   computername/
      |
      duetip/
         |
         |processid_jobname/         #_jobname if available

The interpretation is:
Starting from the basedir:

(1) For each computer that DuetLapse3 is running on there will be a separate directory(computername). Technically the computername will be the fully qualified domain name (FQDN). In any case - each computer needs to (and in any case should) have a unique FQDN

(2) Underneath the computername directory there will be a separate directory for each Duet that computer is connected to (duetip). All videos for this computer and duet combination go into this directory as well as the respective logfiles.

(3) Underneath the duetip directory will be temporary directories (processid_jobname).If the printjob has not started at that time - there will be no _jobname portion. This handles the the situation where multiple instances of DuetLapse3 are running on the same computer against the same Duet. This directory is created when the first image is captured.

Note that the filenames have been rationalized for this new structure. Videos and logfiles are prefixed by the processid.

To provide cross platform compatibility, colons are replaced by raised colons (more-or-less look the same). Spaces in filenames are replaced by underscore.

This is a many to many possibility that few will need but it's future proofed. This way there should be no collisions even when stored on a shared drive.

The following logic will be used to clean up files more-or-less automatically.
On startup:
[1] DuetLapse will determine how many instances are running on that computer and note their processid's. It will then delete all of the processid directories it finds (in (basedir/computername/duetip) that do not match. These will be from older (not running) instances. i.e. it will delete the old image files.
[2] If -keeplogs is NOT used. The FIRST instance of DuetLapse3 (with no other instances running) will delete the old logfiles. If -keeplogs is used then the logfiles will not be deleted. This means that if you typically use -keeplogs you can affect a cleanup by running a single instance of DuetLapse3 without -keeplogs.

On Terminate
if -deletepics is used. The basedir/computername/duetip/processid directory will be deleted (i.e. the images will be deleted).

The new option -novideo will prevent a video being created when DuetLapse3 terminates. If BOTH -novideo and -deletepics are specified- this means that if you want to use the images you need to have done so before terminating that instance of DuetLapse3. For example using the snapshot feature provided by the html listener.

[GoremanX]

@stuartofmt Very cool, great job! You just covered all of the complexities of my crazy use-case, and that's no small feat I look forward to trying it out as soon as my warranty replacement Duet 3 board arrives... which should be any month now...

[JohnOCFII]

I'm looking forward to trying this new version. I haven't moved beyond the original yet, and I've got dying process issues on my Pi that might be related to how I hacked that together.

John

[stuartofmt]

@JohnOCFII said in DuetLapse3:

I'm looking forward to trying this new version. I haven't moved beyond the original yet, and I've got dying process issues on my Pi that might be related to how I hacked that together.

John

I have put in quite a lot of defensive code compared to the original. So this version should, in general, be more robust. As a quick datum - last night I ran three instances concurrently on a Pi 3B+ each with a browser tab polling status every 60 seconds and as well startDuetLapse3 running. Each instance captured images every 60 seconds from the camera (also running on the Pi). So a reasonable "soak test". Each captured over 750 images (~ 2200 in total).

I then issued a "terminate all" from startDuetLapse3 and only two videos were successfully created. The problem was an out-of-memory error from ffmpeg. I've not seen this type of error before and guess it was three instances of ffmeg trying all at once to create a video. I'm going to try and increase the swapfile to see if that changes anything. The images are all there - so it is recoverable.

As an aside I have startDuetLapse3 running under systemctl with restart - so it is continuously running and will restart after a shutdown - so its "always there"

[stuartofmt]

@stuartofmt said in DuetLapse3:
I'm going to try and increase the swapfile to see if that changes anything. The images are all there - so it is recoverable.

I increased the swapfile and that does seem to have solved the issue. Success with 3 instances, each with over 1300 images (total ~ 4000). The Pi I tested on only had 100M of swapfile. I increased it to 1024M. During the test ~ 350M was used. I'm on Debian Buster so the change was an edit to /etc/dphys-swapfile followed by a reboot. I did check but there does not seem to be a simple way to limit memory usage in ffmpeg - so this looks like the way to go.

[GoremanX]

@stuartofmt Looks like I need to update the master image for my Pis! Out of curiosity, how much memory is on the boards you've been testing with? I'm a cheap bastard, I tend to stick to the base boards, so all mine have 1GB (older) or 2GB (current). I've been considering getting some fancy-schmancy 8GB boards for the next ones, but it's hard to justify that $40 premium just for memory I might never make use of

[stuartofmt]

@GoremanX said in DuetLapse3:

Out of curiosity, how much memory is on the boards you've been testing with?

From

cat /proc/cpuinfo | grep Model

Model                 : Raspberry Pi 3 Model B Rev 1.2

This model has:
1.2 GHz 64-bit quad core ARM Cortex-A53 and 1GB RAM
I have a 32GB card.
In all - it seems to cope well. Here is what it looks like with the following load:

• Camera streaming - 9 simultaneous connections (3 internal - 6 external taking snapshots ~ 2 per second) + 1 external constantly streaming connection.
• startDuetLap connectionse3
• DuetLapse3 - 3 instances capturing images every 5 seconds
• chromium - running dueui

Chromium is the biggest load ~ 17%. ffmpeg slips in periodically at about 12% and the DuetLapse3 instances each peak at 7% when they are busy .

The load average suggests this is handling it pretty well (lets say ~ 25% CPU utilization).

But as you can see - the previous 100M swapfile was not enough. 1GB may be a little big but it seems like it will be a "safe" number.

This is further supported by:

pi:~ $ uptime
 11:15:16 up 18:58,  3 users,  load average: 1.21, 1.08, 0.98

[GoremanX]

Finally got a chance to try this yesterday. It works perfect. I'm using reprap_notify to monitor the state of the printer. It polls the Duet 3 board every 30 seconds. When it detects that a print job has started, it starts an instance of DuetLapse3. When the job ends, it sends a terminate command via curl. I get an email notification of each event, and also if/when the printer pauses. All the files end up exactly where they need to. I have a second browser tab for monitoring the status of DuetLapse3. Everything appears to run reliably.

[stuartofmt]

@GoremanX said in DuetLapse3:

It works perfect.

Thanks for the feedback!
I'm sure you have not tried hard enough to break it

I do have a small update coming but this is just to protect against some ffmpeg limitations. ffmpeg tries to gobble up lots of CPU and Memory if multiple instances are running. Does not affect most users but surfaced in my stress testing.

[stuartofmt]

@GoremanX said in DuetLapse3:

When it detects that a print job has started, it starts an instance of DuetLapse3.

I'm curious as to the utility of this approach versus DuetLapse3 just sitting there waiting for a print job to start (-detect layer) ? I'm asking as it may suggest missing functionality or improvements.

@GoremanX said in DuetLapse3:

When the job ends, it sends a terminate command via curl.

DuetLapse3 will terminate itself when the print job completes - so wondering why you are sending a separate command ? A bug in a prior release ?

[GoremanX]

@stuartofmt I found that the layer detection starts capture too late sometimes. I prefer to use "dontwait" and start the process manually, even if it results in capturing initial Calibration and bed probing for nothing. I'd rather have too many photos at the beginning than not enough.

As for terminating manually, that's mostly to be 100% sure all processes got terminated. I send pid=all because in my case, a specific pi only ever monitors a single printer at a time. But having said that, I've also found that capturing ends a tad too soon. I never get a capture with the print head out of frame once the print is complete, which would be nice to have as the final frame of the video. Rather than end at the end of the last layer, it would be nice if it ended once the job is actually complete and took at least one last photo at that point before terminating

[stuartofmt]

@GoremanX said in DuetLapse3:

@stuartofmt I found that the layer detection starts capture too late sometimes.

Thanks for pointing that out. I may have a bug there (off by one layer). I'll take a closer look.

..... I've also found that capturing ends a tad too soon. ...... it would be nice if it ended once the job is actually complete .....

Again - thanks for pointing this out.

The Duet transitions to "idle" when the print job is complete. I can't be totally sure what it considers "complete". The last of the gcode command enqueued? The last movement completed? Something in between?
@dc42 any clarity you can provide would be appreciated.

In any case the problem is likely that there is no "last layer" transition. I.e. the start of the last layer is captured but not the end. One way around this might be to include end of job gcode to send the print head to X0,Y0 (G0 X0 Y0) followed by a relative move of the Z axis (G91 G0 Z1).

Another may be to force one last capture when idle is detected - but who knows where the head would be ?

I'll play around over the next few days and see if anything becomes clearer.

[stuartofmt]

@GoremanX

I took a look at my existing gcode. This explains why I had not seen the "ends a tad too soon" phenomenon.

G91           ;relative mode
G0 Z1         ;E Move the bed away a little note use of E after comment for correct layer count
G90           ;Back to absolute mode
G0 X0 Y0      ;Get the print head out of the way

This is what the last frame of my my last print ended up looking like.
(P.S. looks like a E retract may be helpful to eliminate that string )

[GoremanX]

@stuartofmt Ah, I get it. So you've got a move to get the print head out of the way before the end of the gcode file. I'll just add something similar, then, thank you!

[XerxasJade]

Hi,

I recently swapped from Danal's to this version, but I am having issues with the resolution of the images? In the old version I could use -r to force them to be taken at 1920x1080 but for some reason I am stuck with images that are 352x288

The code I am currently running is:
python3 ./DuetLapse/DuetLapse3.py -duet tronxyduet2 -detect pause -extratime 5

I tried to work out the camera parameters for this version myself but I have to admit Linux is not my strongest point. I am running this on a Pi V3

Also possibly related/unrelated when the print finished and it went to make the video, the video came out as 0kb..

For Reference I am running:
RepRapFirmware for Duet 2 WiFi/Ethernet 3.2.2 (2021-02-11)
Duet Web Control 3.1.1

[stuartofmt]

@XerxasJade said in DuetLapse3:

Hi,

I recently swapped from Danal's to this version, but I am having issues with the resolution of the images? In the old version I could use -r to force them to be taken at 1920x1080 but for some reason I am stuck with images that are 352x288

Hi - First let me say that, in terms of the method for creating video's - there is not much difference between Danal older version and mine. The main difference is if extratime is specified.

Basically though - it looks like there are two problems (1) the video not being created and (2) the resolution of the captured images.

For problem 1:
To use extratime on the Pi requires a more modern version of ffmpeg than is normally installed. I call this out in the documentation but if its unclear - let me know. What version are you running ?

ffmpeg -version

also what is the output (last line) of

ffmpeg -filters | grep tpad

It should be something like:
... tpad V->V Temporarily pad video frames.

If it's not there - your version of ffmpeg does not support extratime

Just by coincidence - I ran into the same problem today. I upgraded the Pi yesterday and it replace the more modern version of ffmpeg with an older one
The solution - if you want to use extratime is to follow the notes I supplied in github and to create a newer version .

I just recreated a newer version. It takes a long time (~ an hour) on the Pi - mostly waiting for it to compile. Anyway the video generation works again.

Once you have a new version of ffmpeg - you can recover your "lost" video's (with extratime) by going into the directory with the captured images and execute the following (or similar):

ffmpeg -r 10 -i "Camera1_%08d.jpeg" -c:v libx264 -vf tpad=stop_mode=clone:stop_duration=5  test.mp4

or if you just want to create a video with your current version of ffmpeg - try:

ffmpeg -r 10 -i "Camera1_%08d.jpeg" -vcodec libx264 -y test.mp4

Problem 2:
The standard image capture on DuetLapse3 (and Danals version) does not set image resolution. You did not say but I'm assuming you are using the Pi camera? If this is the case the defaut is full resolution . In general the image resolution is done by whatever application is capturing the images. What are you using ? I'm unclear what you mean by setting -r ?

The -r option on ffmpeg affects the frame rate (i.e. speed of timelapse) but not the resolution.

Bottom line - need some more information before we can solve the resolution issue but I'm pretty sure it lies outside DuetLapse3.