pydsfapi [v3.2.0] - Official Python Client Library for DSF

Danal

@gtj0 said in pydsfapi - Official Python Client Library for DSF [beta]:

If @chrishamm doesn't show up soon, I think I'm just going to rewrite the entire DSF in python (or nodejs).

I would actually vote for NodeJS, but either would be OK.

Danal

P.S. I did fix the M999 bug. Caveat Emptor and all that, this is my very first .net on Linux... etc, etc. If anyone is willing to try the 1.2.4.0 version of DuetControlServer with this fix, download this file:

http://danalspub.com/wp-content/uploads/2020/03/DuetControlServer.zip

Unzip, and place the two files inside in /opt/dsf/bin

The 'DuetControlServer' file should have attributes of -rwxr-xr-x. If for some reason it does not, run chmod 755 DuetControlServer

After the copies:
sudo systemctl stop duetcontrolserver
sudo systemctl start duetcontrolserver

If you decide to go back, run:

sudo apt-get reinstall duetcontrolserver

or

sudo apt-get reinstall duetcontrolserver=specific version number you wish

You can find all available versions via:
apt-cache policy duetcontrolserver

wilriker

@gtj0 said in pydsfapi - Official Python Client Library for DSF [beta]:

If @chrishamm doesn't show up soon, I think I'm just going to rewrite the entire DSF in python (or nodejs).

He's currently working on both DWC and DSF to get the ObjectModel up to date.

But either way please don't replace one interpreted language for another. On a RPi use something that runs natively. Use Go or Rust.

gtj0

@wilriker said in pydsfapi - Official Python Client Library for DSF [beta]:

@gtj0 said in pydsfapi - Official Python Client Library for DSF [beta]:

If @chrishamm doesn't show up soon, I think I'm just going to rewrite the entire DSF in python (or nodejs).

He's currently working on both DWC and DSF to get the ObjectModel up to date.

But either way please don't replace one interpreted language for another. On a RPi use something that runs natively. Use Go or Rust.

HA!. I hear COBOL is making a comeback. Actually the new "thing" is nim

wilriker

@Danal (and interested others):

I have pushed a change that has added an examples.py that shows minimal examples of the three (four) most common usage scenarios.

Also mind that I renamed the main entry point from a generic connections.py to a more specific pydsfapi.py.

Danal

Thank you! I will take a look, and target adapting this into my tool alignment scrips.

JBisc

@wilriker For beginners (of APIs) it would be very useful, so see an example in which you interact more with the duet (like move/ call M115 / read pin). I feel slighly overwhelmed with your example....

wilriker

@JBisc There is an update to pydsfapi coming in the next couple of days. I'll update the examples then also. Thanks for pointing it out.

wilriker

@JBisc I haven't changed much with the examples but added a lot of comments (see https://github.com/Duet3D/DSF-APIs/blob/dev/pydsfapi/examples.py). Is that already enough to get you started? If not can you please provide a detailed set of things you want to achieve and I'll be happy to either add this to the examples.py or reply to it here in the thread.

JBisc

@wilriker Thanks. I will try it directly tomorrow and give feedback to you. Just for others who are searching: these changes are still on the dev Branch.

JBisc

@wilriker Ok I tried. I took the latest commit on the dev branch (Hash: 7b7c9291aaa1e77bd570c2503f981a73410a0f32)

I changed:

cmd_conn.connect('/var/run/dsf/dcs.sock')

I ran the example file with python 3.7 and the RPI and sudo rights (dcs.sock requires it)

and I got the following error

File "/home/shares/pi/HandwritingRoboter/src/DSF-APIs/pydsfapi/pydsfapi.py", line 234, in connect
  server_init_message.EXPECTED_SERVER_VERSION, server_init_message.version))
AttributeError: 'ServerInitMessage' object has no attribute 'EXPECTED_SERVER_VERSION'

Probably you have an Idea what that could be?

wilriker

@JBisc Yeah, the reason was me refactoring a name right before committing but after testing.

But there is another problem why it would not have happened for me anyway: the version on the dev branch requires DSF 2.1.0 from unstable repo.

EDIT: I pushed another change fixing this bug right before I posted here. So once you upgraded to DSF 2.1.0 the latest commit on dev should work.

wilriker

Release 2.1.0

I present to you the first official release of pydsfapi which can be found at GitHub Releases page as usual.

This is the first release of pydsfapi and it is target at compatibility with Duet Software Framework 2.1.0 and later (it specifically does not work with DSF 1.2.4 and earlier).

With this release the package structure has been brought to a Python standard layout and a setup.py has been added for easier installation via disttools.

What's missing

There are no concrete class implementations representing the ObjectModel. For now there is a placeholder class MachineModel that has all the values as dict with dicts inside it.

wilriker

Release v2.1.2

This release can be found as usual on GitHub Releases page.

Port changes from upstream

Increase PROTOCOL_VERSION to 6
Add G53 to string output of commands that use it

wilriker

Release 3.1.0

This release syncs changes and version with latest DuetSoftwareFramework and can be found as usual on GitHub Releases page.

Ported changed from upstream

Increase PROTOCOL_VERSION to 7
Add new Aux2 CodeChannel
Comments now use Q as CodeType (instead of C)
Letter for unprecedented parameters has been changed from empty string to @

wilriker

Release 3.1.1

Minor release that just updates PROTOCOL_VERSION to 8 to be in line with DuetSoftwareFramework 3.1.1.

Can be found at GitHub Releases page.

achrn

Line 284 of pydsfapi/pydsfapi.py limits the json response to 32kB:
json_string = self.socket.recv(32 * 1024).decode('utf8')

The full machine model can easily exceed that when a print is under way, because it contains an entry for every layer, with at least 90 bytes per layer (depending on the values and number of filaments). Then the json string is just arbitrarily truncated at 32kB and you get an error such as:

Traceback (most recent call last):
  File "./delta_lights.py", line 99, in <module>
    mm = subscribe_connection.get_machine_model()
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 455, in get_machine_model
    machine_model = self.receive(machinemodel.MachineModel)
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 275, in receive
    return cls.from_json(json.loads(json_string))
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 32768 (char 32767)

If you change the (32 * 1024) to something bigger it goes away, but it doesn't take a very large print to overwhelm any sensible value - 32kB at 90 bytes per layer is only 364 layers, so another 32kB gets you less than a further 73mm of print height at 0.2mm layers.

As I understand it, there's the facility in the DSF to filter the machine model returned. However, I haven't figured out of that is in pydsfapi and if so how to use it. Can a filter be used to not return the 'layers' entries?

wilriker

@achrn Thanks for reporting. I will look into that.

achrn

@wilriker
With the self.socket.recv(32 * 1024) changed to self.socket.recv(64 * 1024) and a routine interrogating the machine model and doing some occasional command_connection.perform_simple_code('M573 P0') at the same time, I've had some prints stop part-way (but neglected to record the error in detail - at the time I thought I knew what had done it so thought I didn't need to pay attention to what it was saying).

This may be coincidence, though it's the only time the printer has stopped mid-print, but it may be that putting that receive up to 64 kB upsets something (possibly just takes too long).

ofliduet

I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"

causing a parser error

  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
    return super().connect(sim, socket_path)
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)

Is that a known problem?

I'm also wondering about whether the API is threadsafe. A complete worked example with a thread running in the background periodically updating a complete machine model with patches would be extremely helpful.