pydsfapi [v3.2.0] - Official Python Client Library for DSF

gtj0

@wilriker said in pydsfapi - Official Python Client Library for DSF [beta]:

@gtj0 said in pydsfapi - Official Python Client Library for DSF [beta]:

If @chrishamm doesn't show up soon, I think I'm just going to rewrite the entire DSF in python (or nodejs).

He's currently working on both DWC and DSF to get the ObjectModel up to date.

But either way please don't replace one interpreted language for another. On a RPi use something that runs natively. Use Go or Rust.

HA!. I hear COBOL is making a comeback. Actually the new "thing" is nim

wilriker

@Danal (and interested others):

I have pushed a change that has added an examples.py that shows minimal examples of the three (four) most common usage scenarios.

Also mind that I renamed the main entry point from a generic connections.py to a more specific pydsfapi.py.

Danal

Thank you! I will take a look, and target adapting this into my tool alignment scrips.

JBisc

@wilriker For beginners (of APIs) it would be very useful, so see an example in which you interact more with the duet (like move/ call M115 / read pin). I feel slighly overwhelmed with your example....

wilriker

@JBisc There is an update to pydsfapi coming in the next couple of days. I'll update the examples then also. Thanks for pointing it out.

wilriker

@JBisc I haven't changed much with the examples but added a lot of comments (see https://github.com/Duet3D/DSF-APIs/blob/dev/pydsfapi/examples.py). Is that already enough to get you started? If not can you please provide a detailed set of things you want to achieve and I'll be happy to either add this to the examples.py or reply to it here in the thread.

JBisc

@wilriker Thanks. I will try it directly tomorrow and give feedback to you. Just for others who are searching: these changes are still on the dev Branch.

JBisc

@wilriker Ok I tried. I took the latest commit on the dev branch (Hash: 7b7c9291aaa1e77bd570c2503f981a73410a0f32)

I changed:

cmd_conn.connect('/var/run/dsf/dcs.sock')

I ran the example file with python 3.7 and the RPI and sudo rights (dcs.sock requires it)

and I got the following error

File "/home/shares/pi/HandwritingRoboter/src/DSF-APIs/pydsfapi/pydsfapi.py", line 234, in connect
  server_init_message.EXPECTED_SERVER_VERSION, server_init_message.version))
AttributeError: 'ServerInitMessage' object has no attribute 'EXPECTED_SERVER_VERSION'

Probably you have an Idea what that could be?

wilriker

@JBisc Yeah, the reason was me refactoring a name right before committing but after testing.

But there is another problem why it would not have happened for me anyway: the version on the dev branch requires DSF 2.1.0 from unstable repo.

EDIT: I pushed another change fixing this bug right before I posted here. So once you upgraded to DSF 2.1.0 the latest commit on dev should work.

wilriker

Release 2.1.0

I present to you the first official release of pydsfapi which can be found at GitHub Releases page as usual.

This is the first release of pydsfapi and it is target at compatibility with Duet Software Framework 2.1.0 and later (it specifically does not work with DSF 1.2.4 and earlier).

With this release the package structure has been brought to a Python standard layout and a setup.py has been added for easier installation via disttools.

What's missing

There are no concrete class implementations representing the ObjectModel. For now there is a placeholder class MachineModel that has all the values as dict with dicts inside it.

wilriker

Release v2.1.2

This release can be found as usual on GitHub Releases page.

Port changes from upstream

Increase PROTOCOL_VERSION to 6
Add G53 to string output of commands that use it

wilriker

Release 3.1.0

This release syncs changes and version with latest DuetSoftwareFramework and can be found as usual on GitHub Releases page.

Ported changed from upstream

Increase PROTOCOL_VERSION to 7
Add new Aux2 CodeChannel
Comments now use Q as CodeType (instead of C)
Letter for unprecedented parameters has been changed from empty string to @

wilriker

Release 3.1.1

Minor release that just updates PROTOCOL_VERSION to 8 to be in line with DuetSoftwareFramework 3.1.1.

Can be found at GitHub Releases page.

achrn

Line 284 of pydsfapi/pydsfapi.py limits the json response to 32kB:
json_string = self.socket.recv(32 * 1024).decode('utf8')

The full machine model can easily exceed that when a print is under way, because it contains an entry for every layer, with at least 90 bytes per layer (depending on the values and number of filaments). Then the json string is just arbitrarily truncated at 32kB and you get an error such as:

Traceback (most recent call last):
  File "./delta_lights.py", line 99, in <module>
    mm = subscribe_connection.get_machine_model()
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 455, in get_machine_model
    machine_model = self.receive(machinemodel.MachineModel)
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 275, in receive
    return cls.from_json(json.loads(json_string))
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 32768 (char 32767)

If you change the (32 * 1024) to something bigger it goes away, but it doesn't take a very large print to overwhelm any sensible value - 32kB at 90 bytes per layer is only 364 layers, so another 32kB gets you less than a further 73mm of print height at 0.2mm layers.

As I understand it, there's the facility in the DSF to filter the machine model returned. However, I haven't figured out of that is in pydsfapi and if so how to use it. Can a filter be used to not return the 'layers' entries?

wilriker

@achrn Thanks for reporting. I will look into that.

achrn

@wilriker
With the self.socket.recv(32 * 1024) changed to self.socket.recv(64 * 1024) and a routine interrogating the machine model and doing some occasional command_connection.perform_simple_code('M573 P0') at the same time, I've had some prints stop part-way (but neglected to record the error in detail - at the time I thought I knew what had done it so thought I didn't need to pay attention to what it was saying).

This may be coincidence, though it's the only time the printer has stopped mid-print, but it may be that putting that receive up to 64 kB upsets something (possibly just takes too long).

ofliduet

I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"

causing a parser error

  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
    return super().connect(sim, socket_path)
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)

Is that a known problem?

I'm also wondering about whether the API is threadsafe. A complete worked example with a thread running in the background periodically updating a complete machine model with patches would be extremely helpful.

ofliduet

Different question: Is it possible to use the API to access the socket from a different host? It looks so close to me but no catch.

achrn

@ofliduet
Could it be the same issue, but truncating the json in a different location triggering a different error in the json decoder? That is, mine was failing because the truncation happened part-way through a quoted string, if you're not part-way through a string, presumably you'd get a different sort of json decode error?

One diagnostic would be to print something with very few layers (one or two) to reset the layers dict in the machine model, then try again.

Also, when I was doing stuff initially I sometimes got an error that if I simply repeated the same script it didn't error. I didn't get to the bottom of that, but one thing I'd try is run the same script again and see if it works.

ofliduet

@achrn Don't think so. This is an initial model, coming in unsolicited directly after the connection has been made. The printer is standing still and I can see the complete model in the recv line. The problem might actually be on the dsf side, where it shouldn't send out a machine model unless specifically requested.