DuetLapse3

stuartofmt

    referer = self.headers['Host']
    split_referer = referer.split(":", 1)

What this is saying is that the request (from the browser) did not contain a well formed header url (what you would normally see in the first part of the address bar) E.g. localhost:port ..... or ipaddress:port ......
Just why I cannot say.

Does it generally work ? Was this a transient ? Is it repeatable in any way ?

I'm about to release a small upgrade and might be able to include some checks / mitigation but don't really know what to do if it fails. Any verbose logs or other info would be especially helpful.

JohnOCFII

@stuartofmt Greetings!

I added -verbose and ran from the command line with no issues. So -- I added verbose into the systemctl script, and it ALSO ran without issue. Then I removed the -verbose tag, and things continued to work.

I blame it on the Moderna vaccine I received the other day.

I have no idea what my issue was... Sorry if I caused you to rack your brain.

John

pi@octocore:/opt/DuetLapse $ 
pi@octocore:/opt/DuetLapse $ python3 /opt/DuetLapse/startDuetLapse3.py -port 8082 -args="-duet railcore.localdomain -camera1 web -weburl1 http://octocore.localdomain:8081/snapshot -detect layer -extratime 4 -verbose"
***** Started http listener *****
Cleaning up phase:  startup
railcore.localdomain 
railcore.localdomain ################### Options selected for this run #####################
railcore.localdomain #Environment Settings:
railcore.localdomain # printer         = railcore.localdomain                              
railcore.localdomain # basedir         = /opt/DuetLapse                                    
railcore.localdomain # poll            = 5                                                 
railcore.localdomain # logtype         = both                                              
railcore.localdomain # verbose         = True                                              
railcore.localdomain # os              = Linux                                             
railcore.localdomain # host            = 0.0.0.0                                           
railcore.localdomain # port            = 0                                                 
railcore.localdomain # pid             = 5350                                              
railcore.localdomain # keeplogs        = False                                             
railcore.localdomain # novideo         = False                                             
railcore.localdomain # deletepics      = False                                             
railcore.localdomain # maxffmpeg       = 2                                                 
railcore.localdomain #Execution Setings:
railcore.localdomain # dontwait        = False                                             
railcore.localdomain # seconds         = 0                                                 
railcore.localdomain # detect          = layer                                             
railcore.localdomain # pause           = no                                                
railcore.localdomain # standby         = False                                             
railcore.localdomain #Camera1 Settings:
railcore.localdomain # camera1         = web                                               
railcore.localdomain # weburl1         = http://octocore.localdomain:8081/snapshot         
railcore.localdomain # Video Settings:
railcore.localdomain # extratime   =     4.0                                               
railcore.localdomain ###################################################################
railcore.localdomain 
ffmpeg version git-2020-12-24-29cef1b Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --extra-cflags=-I/usr/local/include --extra-ldflags=-L/usr/local/lib --extra-libs='-lpthread -lm -latomic' --arch=armel --enable-gmp --enable-gpl --enable-libass --enable-libdrm --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libssh --enable-libvorbis --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-mmal --enable-nonfree --enable-omx --enable-omx-rpi --enable-version3 --target-os=linux --enable-pthreads --enable-openssl --enable-hardcoded-tables
  libavutil      56. 62.100 / 56. 62.100
  libavcodec     58.115.102 / 58.115.102
  libavformat    58. 65.100 / 58. 65.100
  libavdevice    58. 11.103 / 58. 11.103
  libavfilter     7. 94.100 /  7. 94.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
 ... tpad              V->V       Temporarily pad video frames.
railcore.localdomain 
railcore.localdomain ###############################################################
railcore.localdomain Connected to printer at railcore.localdomain using Duet version 3.2.2 and API access using rr_model
railcore.localdomain ###############################################################
railcore.localdomain 
railcore.localdomain 
railcore.localdomain ##########################################################
railcore.localdomain Will start capturing images on first layer change
railcore.localdomain ##########################################################
railcore.localdomain 
railcore.localdomain 
railcore.localdomain ##########################################################
railcore.localdomain Video will be created when printing ends.
railcore.localdomain On Linux   -  press Ctrl+C one time to stop capture and create video.
railcore.localdomain On Windows -  Ctrl+C will NOT create a video. You must use the http listener
railcore.localdomain ##########################################################
railcore.localdomain 
railcore.localdomain ++++++ Entering start state ++++++
railcore.localdomain ++++++ Entering start state ++++++
railcore.localdomain ****** Duet status changed to: idle *****
railcore.localdomain ****** Print State changed to: Capturing *****
railcore.localdomain !!!!!! Stopped by SIGINT or CTL+C - Post Processing !!!!!!
railcore.localdomain ++++++ Entering terminate state ++++++
railcore.localdomain Working directory created at: /opt/DuetLapse/octocore/railcore-localdomain/5350
2021-03-28 15:02:21 URL:http://octocore.localdomain:8081/snapshot [128008/128008] -> "/opt/DuetLapse/octocore/railcore-localdomain/5350/Camera1_00000001.jpeg" [1]
railcore.localdomain Camera1: Cannot create video with only 1 frames
Cleaning up phase:  terminate
railcore.localdomain Program Terminated
^C!!!!!! Stopped by SIGINT or CTL+C  !!!!!!
Killed
pi@octocore:/opt/DuetLapse $

stuartofmt

Some fairly small but useful additions. Personally - I am tending to use -keepfiles in all instances and using startDuetLapse3 to manually tidy up etc. as I need to.

###Version 3.4.2###
Changes to DuetLapse3
-[1] Added a new options: -keepfiles to prevent file deletion on startup and shutdown.
-[2] Terminate (from the UI) now offers two options: Graceful and Forces. Graceful is the same as in prior versions. Forced does a quick shutdown with no image capture.
-[3] Added an optional argument (-maxffmpeg) that limits the number of concurrent ffmpeg instances. Ffmpeg can fail due to lack of resources - the default is 2 instances.
Note that this only applies to video creation. Image capture, because it is transient, is not limited.

Changes to startDuetLapse3
[1] Can now delete empty directories (provided they are not in use). This allows a complete cleanup of the directory tree.
[2] Added file function to create a Video on directories containing jpeg files (provided they are not in use).
[3] Added an optional argument (-maxffmpeg) that limits the number of concurrent ffmpeg instances. Ffmpgeg can fail due to lack of resources - the default is 2 instances.
Note that this only applies to video creation.

war4peace

Hi all,

I have installed DuetLapse3 and verified it's running well (it does).
But with me being a newbie at all Linux-things, I'm not sure about some basic things...

What would be the best way to have DuetLapse3 start at boot? I have added this line to rc.local but not sure if it would work:

/home/pi/duetlapse/duetlapse.sh start &

If I run "duetlapse.sh start" it works.

Is it ok if I keep DuetLapse3 running at all times? Would it start capturing automatically, every time I start a print, and stop capturing when print is finished? I have DWC, uStreamer and DuetLapse3 on the same RPi (RPi 4 4 GB).

3 (LE). One more thing. I have noticed that DuetLapse3 terminates itself after a print is finished and a video is created. I would like to automate video capture and not worry about whether DuetLapse3 is started or not when I start a job. This means I'm fine with DuetLapse3 being active all the time and capturing all my prints. Question is, how would I go about automating that? Keep DuetLapse3 idle and waiting all the time, start capturing when print starts, save video and go back to idle, waiting for a new job to start.

JohnOCFII

Greetings -- My last couple of prints, I noticed that I wasn't getting my "extratime". The log told me that my version of ffmpeg didn't support it, even though it was the version I built locally on my Pi last December.

pi@octocore:/opt/DuetLapse/octocore/railcore-localdomain $ more 15379_21-05-17T16˸40˸44.log 
2021-05-17 16:40:45,112 - 
2021-05-17 16:40:45,114 - ################### Options selected for this run #####################
2021-05-17 16:40:45,114 - #Environment Settings:
2021-05-17 16:40:45,115 - # printer         = railcore.localdomain                              
2021-05-17 16:40:45,116 - # basedir         = /opt/DuetLapse                                    
2021-05-17 16:40:45,116 - # poll            = 5                                                 
2021-05-17 16:40:45,117 - # logtype         = both                                              
2021-05-17 16:40:45,118 - # verbose         = False                                             
2021-05-17 16:40:45,118 - # os              = Linux                                             
2021-05-17 16:40:45,119 - # host            = 0.0.0.0                                           
2021-05-17 16:40:45,119 - # port            = 0                                                 
2021-05-17 16:40:45,120 - # pid             = 15379                                             
2021-05-17 16:40:45,120 - # keeplogs        = False                                             
2021-05-17 16:40:45,121 - # novideo         = False                                             
2021-05-17 16:40:45,122 - # deletepics      = False                                             
2021-05-17 16:40:45,122 - # maxffmpeg       = 2                                                 
2021-05-17 16:40:45,123 - # keepfiles       = False                                             
2021-05-17 16:40:45,123 - #Execution Setings:
2021-05-17 16:40:45,124 - # dontwait        = False                                             
2021-05-17 16:40:45,125 - # seconds         = 0                                                 
2021-05-17 16:40:45,126 - # detect          = layer                                             
2021-05-17 16:40:45,127 - # pause           = no                                                
2021-05-17 16:40:45,128 - # standby         = False                                             
2021-05-17 16:40:45,128 - #Camera1 Settings:
2021-05-17 16:40:45,129 - # camera1         = web                                               
2021-05-17 16:40:45,130 - # weburl1         = http://octocore.localdomain:8081/snapshot         
2021-05-17 16:40:45,131 - # Video Settings:
2021-05-17 16:40:45,132 - # extratime   =     4.0                                               
2021-05-17 16:40:45,133 - ###################################################################
2021-05-17 16:40:45,133 - 
2021-05-17 16:40:45,432 - 
2021-05-17 16:40:45,433 - ************************************************************************************
2021-05-17 16:40:45,434 - Warning: This version of ffmpeg does not support -extratime
2021-05-17 16:40:45,434 - -extratime has been set to 0
2021-05-17 16:40:45,435 - ************************************************************************************
2021-05-17 16:40:45,758 - 
2021-05-17 16:40:45,759 - ###############################################################
2021-05-17 16:40:45,760 - Connected to printer at railcore.localdomain using Duet version 3.3RC2 and API access using rr_model
2021-05-17 16:40:45,761 - ###############################################################
2021-05-17 16:40:45,761 - 
2021-05-17 16:40:45,762 - 
2021-05-17 16:40:45,763 - ##########################################################
2021-05-17 16:40:45,764 - Will start capturing images on first layer change
2021-05-17 16:40:45,764 - ##########################################################
2021-05-17 16:40:45,765 - 
2021-05-17 16:40:45,766 - 
2021-05-17 16:40:45,767 - ##########################################################
2021-05-17 16:40:45,767 - Video will be created when printing ends.
2021-05-17 16:40:45,768 - On Linux   -  press Ctrl+C one time to stop capture and create video.
2021-05-17 16:40:45,769 - On Windows -  Ctrl+C will NOT create a video. You must use the http listener
2021-05-17 16:40:45,769 - ##########################################################
2021-05-17 16:40:45,770 -

even though the filter showing tpad was there:

pi@octocore:/opt/DuetLapse $ ffmpeg -filters | grep tpad
ffmpeg version git-2020-12-24-29cef1b Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --extra-cflags=-I/usr/local/include --extra-ldflags=-L/usr/local/lib --extra-libs='-lpthread -lm -latomic' --arch=armel --enable-gmp --enable-gpl --enable-libass --enable-libdrm --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libssh --enable-libvorbis --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-mmal --enable-nonfree --enable-omx --enable-omx-rpi --enable-version3 --target-os=linux --enable-pthreads --enable-openssl --enable-hardcoded-tables
  libavutil      56. 62.100 / 56. 62.100
  libavcodec     58.115.102 / 58.115.102
  libavformat    58. 65.100 / 58. 65.100
  libavdevice    58. 11.103 / 58. 11.103
  libavfilter     7. 94.100 /  7. 94.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
 ... tpad              V->V       Temporarily pad video frames.
pi@octocore:/opt/DuetLapse $

I rebuilt ffmpeg again following the instructions in the GitHub pages, but other than seeing that I do have the new version, the log still shows that "Warning: This version of ffmpeg does not support -extratime"

The extra time feature was working fine a couple of months ago.

Any ideas, or places to check?

Thanks,

John

stuartofmt

@johnocfii said in DuetLapse3:

The extra time feature was working fine a couple of months ago.
Any ideas, or places to check?

Let me take a look later today / tomorrow. Its entirely possible that I introduced some logic error in a later version. I did do some 'tidy up' so it could easily 'no good deed goes un-punuished'

What version of DuetLapse3 are you on ? Sorry that you had to rebuild ffmpeg.

stuartofmt

@war4peace said in DuetLapse3:

But with me being a newbie at all Linux-things, I'm not sure about some basic things...

Sorry - for some reason I did not get notification of your post (or I fat fingered it) and only now just saw your question.

I'm tied up all day - but will give you a considered answer later today (if I can ) but more likely tomorrow. The good news is that its all possible and pretty straightforward BUT there are options

JohnOCFII

@stuartofmt said in DuetLapse3:

What version of DuetLapse3 are you on ? Sorry that you had to rebuild ffmpeg.

I'm running 3.4.2. I was on 3.4.1 before that.

John

stuartofmt

@johnocfii said in DuetLapse3:

I'm running 3.4.2. I was on 3.4.1 before that.

Do you mind trying 3.4.3. Looks like it was a silly mistake - but I do not have time to rigorously test at the moment.

JohnOCFII

@stuartofmt sure - I'll try that soon.

JohnOCFII

@stuartofmt Worked great! https://youtu.be/vvtMgD2nNE0

war4peace

@stuartofmt
No rush, I am still tuning and polishing a lot of things with my printer. Take your time

stuartofmt

@war4peace

Hi - I think the simplest approach (for now) is to run DuetLapse3.py using the linux systemctl capability.

I updated the instructions here:
https://github.com/stuartofmt/DuetLapse3/blob/main/systemctl setup.md
and added a sample unit file (.service extension) here:
https://github.com/stuartofmt/DuetLapse3/blob/main/DuetLapse3.service

Note that you want to make sure the options you have set are perfected by running DuetLapse3 from the command console, before you replicate these in the unit file.

Essentially - using this approach, DuetLapse3 will terminate normally at the end of each print job and then restart.

In particular - you likely (definitely) want to have -detect layer set. Otherwise - on restart of DuetLapse3, even if there is no print job running, it may start capturing images the entire time

war4peace

@stuartofmt

Thank you for your answer.

Here's my command line (tested to be working, located in a bash file, see below):

/usr/bin/python3 $PROCESS -duet localhost -basedir /home/pi/duetlapse/output -camera1 stream -weburl1 http://localhost:8181/stream -detect layer -seconds 10  -extratime 10 -instances oneip -deletepics -port 8282 &

I am trying to follow the instructions listed, but I have a couple questions.
In the document it says "This needs to be the directory for BOTH startDuetLapse3.py and DuetLapse3.py" - there is no "startDuetLapse3.py" file to be found.
Here's the full list of files in the DuetLapse folder:

pi@duet3:~/duetlapse $ find "$(pwd)"
/home/pi/duetlapse
/home/pi/duetlapse/duetlapse.sh
/home/pi/duetlapse/DuetLapse3.py
/home/pi/duetlapse/output
/home/pi/duetlapse/output/duet3
/home/pi/duetlapse/output/duet3/localhost
/home/pi/duetlapse/startDuetLapse3.service

As for starting DUetLapse, I am using a bash file with the following contents:

#! /bin/bash
#Example bash script for running DuetLapse3.py

PROCESS=/home/pi/duetlapse/DuetLapse3.py

case "$1" in
   start)
#Edit and uncomment for your own needs
#One camera example
      /usr/bin/python3 $PROCESS -duet localhost -basedir /home/pi/duetlapse/output -camera1 stream -weburl1 http://localhost:8181/stream -detect layer -seconds 10  -extratime 10 -instances oneip -deletepics -port 8282 &
#  Two camera example
#/usr/bin/python3 $PROCESS -duet 192.168.86.235 -basedir /home/pi/Lapse -instances oneip -dontwait -seconds 3 -camera1 stream -weburl1 http://192.168.86.230:8081/stream.mjpg  -camera2 other -weburl2 http://192.168.86.230:8081/st$
   ;;
   stop)
      if ["$2" == '']
      then
          PROCESS_PID=`/bin/ps -ef | grep "$PROCESS" | grep -v "grep" | awk '{print $2}'`
          kill -2 $PROCESS_PID
      else
          PROCESS_PID=`/bin/ps -ef | grep "$PROCESS" | grep "$2" | grep -v "grep" | awk '{print $2}'`
          kill -2 $PROCESS_PID
     fi
   ;;
esac

Can I use that bash script instead of startDuetLapse3.py?

I am a newbie in linux-world, sorry for the arguably stupid questions...

JohnOCFII

@war4peace The file startDuetLapse3.py can be found along with the rest of the DuetLapse files here: https://github.com/stuartofmt/DuetLapse3

It is fairly straightforward to replicate if needed.

John

stuartofmt

@war4peace said in DuetLapse3:

In the document it says "This needs to be the directory for BOTH startDuetLapse3.py and DuetLapse3.py" - there is no "startDuetLapse3.py" file to be found.
.

I'll fix that - It should have read This needs to be the directory in which you have DuetLapse3.py installed. If you are using startDuetLapse3.py it will be in the same directory as DuetLapse3.

Can I use that bash script instead of startDuetLapse3.py?

For your use-case, I would not use a bash script at all. The bash script was there to support a much earlier version, when the documentation was less complete. Since then the inbuilt capability together with startDuetLapse3 makes the bash script redundant (in most cases) . I will remove it as it no longer serves a purpose.

Can I use that bash script instead of startDuetLapse3.py?

startDuetLapse3 is an administrative tool. Its use is optional but has many useful features - way beyond a bash script. It's accessible from a browser - so you can administer from more-or-less any local machine. I also use VPN to access it remotely

If you want a single instance of DuetLapse3 running all the time - then you are better off using systemctl for that instance. The key lines in you unit file (given the info you provided) would look like this.

<snip>
WorkingDirectory=/home/pi/duetlapse
<snip>
ExecStart=python3 /home/pi/duetlapse/DuetLapse3.py -duet localhost -basedir /home/pi/duetlapse/output -camera1 stream -weburl1 http://localhost:8181/stream -detect layer -seconds 10 -extratime 10 -instances oneip -deletepics -port 8282
<snip>

Of course you can run additional ad-hoc instances if you like (e.g. through startDuetLapse3).

With that in mind - you may find -instances oneip too restrictive or redundant. The very first instance of DuetLapse essentially ignores this directive because there can be no conflict. Subsequent instances check already running instances for possible conflict based on their directive only. In other words these options are not global in their scope. With -instances oneip - you cannot have 2 or more instances connected to the same Duet. With -instances many, you can have many instances connected to the same (or different) Duets.

war4peace

@johnocfii

Thank you. I have followed the instructions, but receiving an error, see below:

pi@duet3:~/duetlapse $ sudo systemctl status startDuetLapse3.service
● startDuetLapse3.service - startDuetLapse3 Service
   Loaded: loaded (/lib/systemd/system/startDuetLapse3.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2021-05-21 00:16:32 EEST; 6s ago
  Process: 15508 ExecStart=/usr/bin/python3 startDuetLapse3.py -duet localhost -basedir /home/pi/duetlapse/output -camera1 stream -weburl1 http://localhost:8181/stream -detect layer -seconds 10 -extratime 10 -instances oneip -deletepics -port 8282 & (code=exited, status=2)
 Main PID: 15508 (code=exited, status=2)

May 21 00:16:32 duet3 systemd[1]: startDuetLapse3.service: Service RestartSec=100ms expired, scheduling restart.
May 21 00:16:32 duet3 systemd[1]: startDuetLapse3.service: Scheduled restart job, restart counter is at 5.
May 21 00:16:32 duet3 systemd[1]: Stopped startDuetLapse3 Service.
May 21 00:16:32 duet3 systemd[1]: startDuetLapse3.service: Start request repeated too quickly.
May 21 00:16:32 duet3 systemd[1]: startDuetLapse3.service: Failed with result 'exit-code'.
May 21 00:16:32 duet3 systemd[1]: Failed to start startDuetLapse3 Service.

stuartofmt

@war4peace said in DuetLapse3:

Thank you. I have followed the instructions, but receiving an error, see below:

Just to be clear: Are you intending to start startDuetLapse3.py or DuetLapse3.py?
They are two separate programs startDuetLapse3 provides an administrative, browser-based UI for starting and managing DuetLapse3.py.

In any case:

You did not provide the full path name in the ExecStart command.
It looks like you want to start DuetLapse3.py. If that's the case - you are calling the wrong program.
If you intended to start startDuetLapse3.py then the command options have a different syntax due to some vagaries of how such things are passed by the operating system. See the documentation on startDuetLapse3 here (in particular the -args syntax):
https://github.com/stuartofmt/DuetLapse3/blob/main/startDuetLapse3.md

war4peace

@stuartofmt

Thank you again for your patience.

Updated command line:

ExecStart=/usr/bin/python3 /home/pi/duetlapse/startDuetLapse3.py -duet localhost -basedir /home/pi/duetlapse/output -camera1 stream -weburl1 http://localhost:8181/stream -detect layer -seconds 10  -extratime 10 -instances oneip -deletepics -port 8282

Current status: service starts, then ends after a short while. Running "sudo systemctl status startDuetLapse3.service" immediately after starting the service shows it as running, but after a few seconds it exits. Must be something in the command line.

I am now going through the manual for StartDuetLapse and will adjust accordingly.
Will post updates once done.

EDIT: It is working now.

Thank you very much for your help, much appreciated!

stuartofmt

@war4peace said in DuetLapse3:

I am now going through the manual for StartDuetLapse and will adjust accordingly.

You need to have firmly in your mind which program you are dealing with Many of the options (because of the way you specified them) are incorrect for startDuetLapse3. (which is the program you are trying to start). You are using some options that only apply to DuetLapse3. This is wrong.

If you stop startDuetLapse3

sudo systemctl stop startDuetLapse3.service

and then run the command you are trying to use

/usr/bin/python3 /home/pi/duetlapse/startDuetLapse3.py -duet localhost -basedir /home/pi/duetlapse/output -camera1 stream -weburl1 http://localhost:8181/stream -detect layer -seconds 10 -extratime 10 -instances oneip -deletepics -port 8282

you will see what I mean - you will get errors about the options that you are trying to use.