DuetLapse3
-
@GoremanX said in DuetLapse3:
@stuartofmt I found that the layer detection starts capture too late sometimes.
Thanks for pointing that out. I may have a bug there (off by one layer). I'll take a closer look.
..... I've also found that capturing ends a tad too soon. ...... it would be nice if it ended once the job is actually complete .....
Again - thanks for pointing this out.
The Duet transitions to "idle" when the print job is complete. I can't be totally sure what it considers "complete". The last of the gcode command enqueued? The last movement completed? Something in between?
@dc42 any clarity you can provide would be appreciated.In any case the problem is likely that there is no "last layer" transition. I.e. the start of the last layer is captured but not the end. One way around this might be to include end of job gcode to send the print head to X0,Y0 (G0 X0 Y0) followed by a relative move of the Z axis (G91 G0 Z1).
Another may be to force one last capture when idle is detected - but who knows where the head would be ?
I'll play around over the next few days and see if anything becomes clearer.
-
I took a look at my existing gcode. This explains why I had not seen the "ends a tad too soon" phenomenon.
G91 ;relative mode G0 Z1 ;E Move the bed away a little note use of E after comment for correct layer count G90 ;Back to absolute mode G0 X0 Y0 ;Get the print head out of the wayThis is what the last frame of my my last print ended up looking like.
(P.S. looks like a E retract may be helpful to eliminate that string
)
-
@stuartofmt Ah, I get it. So you've got a move to get the print head out of the way before the end of the gcode file. I'll just add something similar, then, thank you!
-
Hi,
I recently swapped from Danal's to this version, but I am having issues with the resolution of the images? In the old version I could use -r to force them to be taken at 1920x1080 but for some reason I am stuck with images that are 352x288
The code I am currently running is:
python3 ./DuetLapse/DuetLapse3.py -duet tronxyduet2 -detect pause -extratime 5I tried to work out the camera parameters for this version myself but I have to admit Linux is not my strongest point. I am running this on a Pi V3
Also possibly related/unrelated when the print finished and it went to make the video, the video came out as 0kb..
For Reference I am running:
RepRapFirmware for Duet 2 WiFi/Ethernet 3.2.2 (2021-02-11)
Duet Web Control 3.1.1 -
@XerxasJade said in DuetLapse3:
Hi,
I recently swapped from Danal's to this version, but I am having issues with the resolution of the images? In the old version I could use -r to force them to be taken at 1920x1080 but for some reason I am stuck with images that are 352x288
Hi - First let me say that, in terms of the method for creating video's - there is not much difference between Danal older version and mine. The main difference is if extratime is specified.
Basically though - it looks like there are two problems (1) the video not being created and (2) the resolution of the captured images.
For problem 1:
To use extratime on the Pi requires a more modern version of ffmpeg than is normally installed. I call this out in the documentation but if its unclear - let me know. What version are you running ?ffmpeg -versionalso what is the output (last line) of
ffmpeg -filters | grep tpadIt should be something like:
... tpad V->V Temporarily pad video frames.If it's not there - your version of ffmpeg does not support extratime
Just by coincidence - I ran into the same problem today. I upgraded the Pi yesterday and it replace the more modern version of ffmpeg with an older one
The solution - if you want to use extratime is to follow the notes I supplied in github and to create a newer version .I just recreated a newer version. It takes a long time (~ an hour) on the Pi - mostly waiting for it to compile. Anyway the video generation works again.
Once you have a new version of ffmpeg - you can recover your "lost" video's (with extratime) by going into the directory with the captured images and execute the following (or similar):
ffmpeg -r 10 -i "Camera1_%08d.jpeg" -c:v libx264 -vf tpad=stop_mode=clone:stop_duration=5 test.mp4or if you just want to create a video with your current version of ffmpeg - try:
ffmpeg -r 10 -i "Camera1_%08d.jpeg" -vcodec libx264 -y test.mp4Problem 2:
The standard image capture on DuetLapse3 (and Danals version) does not set image resolution. You did not say but I'm assuming you are using the Pi camera? If this is the case the defaut is full resolution . In general the image resolution is done by whatever application is capturing the images. What are you using ? I'm unclear what you mean by setting -r ?The -r option on ffmpeg affects the frame rate (i.e. speed of timelapse) but not the resolution.
Bottom line - need some more information before we can solve the resolution issue but I'm pretty sure it lies outside DuetLapse3.
-
Hey, Thankyou for the quick response.
I had completely missed the ffmpeg version issue. So it turns out I was running 4.1.6 so I'v updated that and i'll see how it goes on tonight's print.
Regarding the -r and resolution issue here is everything I know...
I am using a logitech 1080p webcam connected via usb - I previously had this exact issue with the old script but was able to fix it with the following command:
./DuetLapse/DuetLapse.py -duet "Hostname" camparms -parms -r 1920x1080
This was based on code discussed by baenwort and arhi from the old thread. For me it solved the issue.
I suspect the issue isn't with the duetlapse software and more likely me getting the syntax wrong, or that I need to set the camera somewhere else now.. But that is the bit i'm not 100% sure how to solve/find. I tried playing around camera1 parameters but didn't have much success.
Edit, I'm not sure what software is capturing the images, but i'm assuming fswebcam is doing it maybe?? this is where my linux knowledge runs out
-
@XerxasJade said in DuetLapse3:
@stuartofmt
I am using a logitech 1080p webcam connected via usb - I previously had this exact issue with the old script but was able to fix it with the following command:./DuetLapse/DuetLapse.py -duet "Hostname" camparms -parms -r 1920x1080
This is an area where I made changes to improve on the flexibility of the origin. This new definition of camparam (and vidparam) allows the user to specify arbitrary programs and switches depending on their camera. Put another way - you are not restricted to the camera types, programs and settings of the four standard types.
From the documentation:
-camparam1="[command]"
If omitted has no default. Used in conjunction with -camera1 to define how the images will be captured.
Note the use of the = and quoting of the command string. Single quotes should be used in the command string when needed.
There are 3 internal variables that can be used weburl (which has the value of weburl1), fn (which is the file for the captured images) , debug (which controls verbose logging)example
-camparam1="'ffmpeg -y -i '+weburl+ ' -vframes 1 ' +fn+debug"This example is the same as if -camera1 stream was used. The value of weburl1 would be substituted for weburl and the output goes the the file specification fn. the results are verbose of not is defermined by the internal variable debug. In general both fn and debug should be used. The use of weburl would depend on the capture method being used.
Notes on the use of -camparam1
The following are the standard commands for reference-camera usb
'fswebcam --quiet --no-banner '+fn+debug-camera pi
'raspistill -t 1 -ex sports -mm matrix -n -o '+fn+debug-camera stream
'ffmpeg -y -i '+weburl+ ' -vframes 1 ' +fn+debug-camera web
'wget --auth-no-challenge -nv -O '+fn+' "'+weburl+'" '+debugSo in your case you will need to specify -camera1 other because you are no longer using the default settings for usb.
You then need to specify -camparam1, in your case based on the default usb adding in your specific changes.
-camparam1="'fswebcam --quiet --no-banner -r 1920x1080 '+fn+debug"resulting in a command line that looks like this:
python3 <your path>/DuetLapse3.py -duet "Hostname" -camera1 other -camparam1="'fswebcam --quiet --no-banner -r 1920x1080 '+fn+debug" -
@stuartofmt said in DuetLapse3:
python3 <your path>/DuetLapse3.py -duet "Hostname" -camera1 other -camparam1="'fswebcam --quiet --no-banner -r 1920x1080 '+fn+debug"
Thankyou so much, I had a feeling I was missing something I just didn't know enough, to actually know where to start looking
I would have never worked that out myself, I was close with a couple of things I tried but would have never thought to add the "-camera1 other"
I'll try it when my current print finishes, Thankyou So Much Your a Legend!!
-
Released version 3.4.1. Much nicer to use the browser UI with some extra "goodies" to avoid the need to remote into your server. Some small functional tweaks based on feedback from the last release.
###Version 3.4.1###
Changes to DuetLapse3
[1] Changed the browser UI to a single page layout.
[2] The file function is restricted to the specific instance.
[3] File functions expanded to allow deletion of video files. startDuetLapse3 has more options.
[4] If using -detect layer capture starts on layer 0 (previously was layer 1).
[5] An additional image is captured immediately before a video is created, independent of other settings.
[6] If the version of ffmpeg does not support -extratime it is ignored.Changes to startDuetLapse3
[1] Changed the browser UI to a single page layout.
[2] Added an optional argument (-topdir) to set the top level directory for file functions. If used - this would normally be set the same as DuetLapse or at the "duetip" level
[3] File functions expanded to allow delete and zip. This is "conservative" - will not allow deletion of files / directories of running instances. Can only zip directories.Note: If it was not obvious - Clicking on the links either opens, displays or downloads the directory / file depending on your browser settings.
-
Finally getting around to replacing Danel's version.
I'm running this on CentOS Stream 8.
I received this error on my first test run. I haven't troubleshot yet, as I've got to head out the door, but looks like a character set/codec issue.
My command line:
python3 /opt/DuetLapse/DuetLapse3.py -duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect none -dontwait -seconds 2 -extratime 4My error message:
pi@octocore:~ $ python3 /opt/DuetLapse/DuetLapse3.py -duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect none -dontwait -seconds 2 -extratime 4 Cleaning up phase: startup Traceback (most recent call last): File "/opt/DuetLapse/DuetLapse3.py", line 1732, in <module> init() File "/opt/DuetLapse/DuetLapse3.py", line 225, in init f_handler = logging.FileHandler(logfilename, mode='w') File "/usr/lib/python3.7/logging/__init__.py", line 1092, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python3.7/logging/__init__.py", line 1121, in _open return open(self.baseFilename, self.mode, encoding=self.encoding) UnicodeEncodeError: 'latin-1' codec can't encode character '\u02f8' in position 61: ordinal not in range(256) pi@octocore:~ $ -
Looking a little more closely, issue seems to be in here:
-weburl1 http://octocore.localdomain:8081/snapshotIf I change weburl1 to weburl I get a different error.
Could this be a permissions issue? I installed DuetLapse3 in /opt. I see it did create subdirectories like so:
pi@octocore:/opt $ tree DuetLapse/ DuetLapse/ |-- DuetLapse3.py `-- octocore `-- railcore-localdomain 2 directories, 1 file pi@octocore:/opt $ -
@JohnOCFII said in DuetLapse3:
If I change weburl1 to weburl I get a different error.
Hi John - We'll get you going.
The first thing is that it's weburl1 (numerical 1 at the end). Putting weburl will result in an error message about an unrecognized argument.The other error is complaining because it looks like your OS is set to latin-1 whereas pretty much any code written today (inclusing DuetLapse3) anticipates utf8. The reason for that particular character (raised colon) is because Windows does not allow a regular colon in filenames and visually - colons are useful for time representation .... so I used raised colons for cross platform consistency
The area in the code where the issue is being raised looks to be quite early - setting up the logfile and I suspect the OS is complaining at that point.
Are you willing to change your Centos settings?
This seems to suggest that its pretty straightforward.
https://serverfault.com/questions/275403/how-do-i-change-my-locale-to-utf-8-in-centosEdit: It could also be the way the filesystem is mounted. Take a look at /etc/fstab - there may be some limiting settings there ....
And this looks like it may be useful:
https://www.osetc.com/en/centos-7-rhel-7-change-the-system-locale.html#:~:text=To show the current locale,βcat %2Fetc%2Flocale.For reference - this is what my Debian systems look like:
pi@srsenderpi:~ $ localectl status System Locale: LANG=en_US.UTF-8 VC Keymap: n/a X11 Layout: us X11 Model: pc105 -
Actually, I lied. I have a bunch of CentOS VMs, but THIS is running on a Raspberry Pi.
pi@octocore:/opt $ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debianBut somehow, the locale is not correct:
pi@octocore:/opt $ localectl status System Locale: LANG=en_US VC Keymap: n/a X11 Layout: gb X11 Model: pc105 pi@octocore:/opt $I'm running through raspi-config now to reset locale.
I'll report back soon! -
@stuartofmt Fixing the locale to be UTF-8 compliant got me past that issue -- thanks!
I also had installed DuetLapse in /opt, and had forgotten to set the ownership to user pi, so I was having some permission issues, but that was easy to spot and to fix.
Basic test went fine. Now on to setting up and testing
startDuetLapse3. -
@JohnOCFII said in DuetLapse3:
Basic test went fine. Now on to setting up and testing
startDuetLapse3.Great to hear !
Note that there are a couple of small differences between DuetLapse3 and the original. Mainly in the more exotic settings. The defaults Omitted options) should result in more-or-less the same behavior.
I think you will like startDuetLapse3. You can then do most everything from a browser without "gymnastics".
Appreciate any and all feedback.
-
@stuartofmt So far, so good. About 8 short prints so far today.
One interesting observation. I use multiple subdirectories for my GCODE files on my printer. Up until this last print, all the DuetLapse tests I've done were from one directory. I just did a small test from another directory (at the same level in the printer GCODE files hierarchy), yet it appears DuetLapse created a separate subdirectory for this job, and created and kept all the JPEGs for this last movie.
Here is the directory tree:
pi@octocore:/opt/DuetLapse $ tree . βββ DuetLapse3.py βββ octocore βββ railcore-localdomain βββ 4974_Camera1_Sun-10-51.mp4 βββ 5343_Camera1_Sun-12-18.mp4 βββ 5868_Camera1_Sun-14-59.mp4 βββ 6261_21-03-21T18ΛΈ12ΛΈ26.log βββ 6261_Calibration βΒ Β βββ HoseSideRingTest-v3_0.2mm_PETG_RC βΒ Β βββ Camera1_00000001.jpeg βΒ Β βββ Camera1_00000002.jpeg βΒ Β βββ Camera1_00000003.jpeg βΒ Β βββ Camera1_00000004.jpeg βΒ Β βββ Camera1_00000005.jpeg βΒ Β βββ Camera1_00000006.jpeg βΒ Β βββ Camera1_00000007.jpeg βΒ Β βββ Camera1_00000008.jpeg βΒ Β βββ Camera1_00000009.jpeg βΒ Β βββ Camera1_00000010.jpeg βΒ Β βββ Camera1_00000011.jpeg βΒ Β βββ Camera1_00000012.jpeg βΒ Β βββ Camera1_00000013.jpeg βΒ Β βββ Camera1_00000014.jpeg βΒ Β βββ Camera1_00000015.jpeg βΒ Β βββ Camera1_00000016.jpeg βΒ Β βββ Camera1_00000017.jpeg βΒ Β βββ Camera1_00000018.jpeg βΒ Β βββ Camera1_00000019.jpeg βΒ Β βββ Camera1_00000020.jpeg βΒ Β βββ Camera1_00000021.jpeg βΒ Β βββ Camera1_00000022.jpeg βββ 6261_Camera1_Sun-18-23.mp4 βββ 812_Camera1_Sat-21-57.mp4 4 directories, 29 files pi@octocore:/opt/DuetLapse $ -
@JohnOCFII said in DuetLapse3:
@stuartofmt So far, so good. About 8 short prints so far today.
One interesting observation. I use multiple subdirectories for my GCODE files on my printer. Up until this last print, all the DuetLapse tests I've done were from one directory.
So it looks like the print job was named /Calibration/HoseSideRingTest-v3_0.2mm_PETG_RC Does this seem right given your situation ? Code-wise, the print job name is appended to the process id and a directory created accordingly.
If this is not the case then I'd need more info to make sense of it - specifically
(1) The options used to start DuetLapse3
(2) The hostname of the computer running DuetLapse3 (looks to be octocore)
(2) the duet printer name (looks to be railcore-localdomain)
(3) the full printjobname and the gcode directory in which it was placed.Are the directories cleaned up when you start a new (only one running) instance of DuetLapse3?
Take a look at the documentation and see if its clear. If not - let me know and I will fix.
EDIT: I can probably (and likely will) suppress the possibility of higher level directory names - but in any case understanding your situation will be helpful.
-
@stuartofmt said in DuetLapse3:
So it looks like the print job was named /Calibration/HoseSideRingTest-v3_0.2mm_PETG_RC Does this seem right given your situation ? Code-wise, the print job name is appended to the process id and a directory created accordingly.
This is correct.
Are the directories cleaned up when you start a new (only one running) instance of DuetLapse3?
Yes -- I think what confused me is that at the time I created that tree, the directory and JPEGs were still around -- I was thinking they got cleaned up at the end of the current job, and not the start of the next job.
I actually prefer your current design.
Also - I am really glad of your current design for another reason -- I love the log file! I ran a 7.5 hour print yesterday evening, and my Pi lost network connectivity 5 times during the print. All the successfully captured images are still available for me to re-run through FFMPEG! Separately I need to try to use the wired Ethernet instead of WiFi to see if the WiFi on this particular Pi is the issue.
-
@JohnOCFII said in DuetLapse3:
I ran a 7.5 hour print yesterday evening, and my Pi lost network connectivity 5 times during the print. All the successfully captured images are still available for me to re-run through FFMPEG! Separately I need to try to use the wired Ethernet instead of WiFi to see if the WiFi on this particular Pi is the issue.
I mostly use a Pi with my Duet2 Wifi. Both are fairly close to the router - so good connectivity. Nonetheless - DuetLapse3 should (provided the connectivity issues do not last too long) work through them. I tested the code by deliberately turning off the Duet , performing multiple emergency restarts etc. In short - with a bit of luck - DuetLapse3 should create the video provided it saw the transition to "idle" at the end of the print job. Note that today I discovered a possible "transient status" that may prevent a video being created at the end of the print job. It's possibly related to my custom ending gcode - but I have a fix
In any case - "yes", that's a good reason for leaving the captured images in place at the end of the job. Also - if you have not noticed - you can use startDuetLapse3 to zip up the directory (after the DuetLapse3 instances has terminated) then download the resultant zipfile, all from a browser.
I'm thinking about adding a "create video" button to simplify the recovery - but at some point - I've go to stop adding in "stuff".
-
@stuartofmt It is great to see how your mind works on this stuff!
FYI -- I did disable WiFi, and switch to wired Ethernet. I am no longer seeing connectivity issues, but Duetlapse is still seeing "something" network related. I will gladly 100% blame this on the weak networking code in the firmware on the Duet2 -- whatever library is used in this hardware (not from dc42 -- but whatever the vendor provides) seems to act as if it hasn't seen changes to networking code since the 1990's.
This is what I see a few times an hour:
railcore.localdomain !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! railcore.localdomain There was a network failure: HTTPConnectionPool(host='railcore.localdomain', port=80): Max retries exceeded with url: /rr_model?key=state.status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x764ad170>: Failed to establish a new connection: [Errno 111] Connection refused')) railcore.localdomain !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! railcore.localdomainor
railcore.localdomain !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! railcore.localdomain There was a network failure: HTTPConnectionPool(host='railcore.localdomain', port=80): Max retries exceeded with url: /rr_model?key=state.status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7664cdf0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) railcore.localdomain !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! railcore.localdomain railcore.localdomain getDuetStatus failed to get data. Code: 9999 Reason: Connection Error railcore.localdomain Printer is disconnected - Trying to reconnect railcore.localdomain ****** Duet status changed to: processing ***** railcore.localdomain ****** Print State changed to: Capturing *****As you suggested, DuetLapse seems to work past it without issue.
John
