pydsfapi [v3.2.0] - Official Python Client Library for DSF

wilriker

@ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:

send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"

causing a parser error

  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
    return super().connect(sim, socket_path)
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)

Is that a known problem?

No it is not. It looks like DCS was sending only a partial response out of unknown reason. Also it did send this unsolicited as you mentioned in a later post. AFAIK it should not do this.
@chrishamm Can you please comment on that part?

I'm also wondering about whether the API is threadsafe. A complete worked example with a thread running in the background periodically updating a complete machine model with patches would be extremely helpful.

pydsfapi is currently not thread-safe. For the time being you either need to synchronize on the connection yourself or create one connection per thread.

wilriker

@achrn said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

Line 284 of pydsfapi/pydsfapi.py limits the json response to 32kB:
json_string = self.socket.recv(32 * 1024).decode('utf8')

Thanks again for reporting this. I fixed it and will release a new version shortly.

wilriker

Release 3.1.2

This is a bugfix release to solve the issue with responses from DCS being capped at 32 KiB.

It can be found as usual on GitHub Releases page.

wilriker

@wilriker said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:

@ofliduet said in pydsfapi [v3.1.1] - Official Python Client Library for DSF:
I'm trying to use the api with a Duet3 board with firmware 3.1.1 (stable) and the initial connect for the subscription gives me the confirmation immediately followed by a complete machine model:
send: {"mode": "Subscribe", "version": 8, "SubscriptionMode": "Patch", "Filter": ""}
recv: {"success":true}{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-19b2","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.1"
causing a parser error
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 446, in connect
    return super().connect(sim, socket_path)
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/home/pi/src/OctoPrint/venv3/lib/python3.7/site-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
Is that a known problem?
No it is not. It looks like DCS was sending only a partial response out of unknown reason. Also it did send this unsolicited as you mentioned in a later post. AFAIK it should not do this.
@chrishamm Can you please comment on that part?

My fault. I was confused. This is as intended from DCS ~~and there is actually still a bug in pydsfapi 3.1.2.~~

EDIT: I revise the latter part. AFAICT it should be fixed now.

achrn

@wilriker Thanks. I've installed and used the new version, though actually my first print job turns out to peak at a 31.9kB machine model, so would have been OK previously (d'oh).

However, I do seem to have provoked the error @ofliduet had:

Traceback (most recent call last):
  File "./mm_time.py", line 12, in <module>
    subscribe_connection.connect()
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 447, in connect
    return super().connect(sim, socket_path)
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 238, in connect
    response = self.receive_response()
  File "/usr/local/lib/python3.7/dist-packages/pydsfapi/pydsfapi.py", line 280, in receive_response
    return json.loads(json_string, object_hook=responses.decode_response)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 17 (char 16)

This was from a really simple:

#!/usr/bin/python3
# subscribe to the machine model and report the timings of most recent print job

import json
import time
from pydsfapi import pydsfapi
from pydsfapi.initmessages.clientinitmessages import SubscriptionMode


# establish the connection
subscribe_connection = pydsfapi.SubscribeConnection(SubscriptionMode.PATCH, debug=False)
subscribe_connection.connect()

try:
    # Get the complete model once
    machine_model = subscribe_connection.get_machine_model()

    [[continues, just prints out some machine model values, omitted for brevity]]

I note that this error is occurring at line 12, before I ask for a copy of the machine model, as @ofliduet reported.

If I do literally nothing but run the function again (i.e. press [up-cursor], press [Enter]) it runs fine. About one run in twenty errors, with no discernible pattern. I've never had it error twice in succession, I have had it error, then run OK twice, then error on the next (I think that's the most frequent repeat I've seen).

wilriker

@achrn Thanks for re-reporting. I know then what the problem is. I just could not reproduce it on my setup.

I already have a solution that I will provide tomorrow. But I need to find a way to make it more efficient. That will be part of the 3.2 release.

wilriker

Release 3.1.3

This is another bugfix release to solve the problem if message in (primarily in Subscription mode) are received faster than they are processed it would lead to JSON parser errors.

It can be found as usual on GitHub Releases page.

Thanks again to @achrn for reporting.

achrn

@wilriker Has the type of the MachineModel type that subscribe_conenction.get_machine_model() returns changed?

My scripts that previously did that then e.g. for h in machine_model['heat']['heaters'] are suddenly reporting TypeError: 'MachineModel' object is not subscriptable.

(This might have been a change in 3.1.2 - I had thought I installed 3.1.2, but when I installed 3.1.3 it suggested it was replacing 3.1.1).

wilriker

@achrn I have noticed something similar but TBH I have no idea what is going on here. Nothing has changed in regards to this but still it is no longer working. I have to investigate that.

Which python version are you using?

achrn

@wilriker Wow that was quick!

It occurred to me I could check for myself when the change happens - it does change between 3.1.1 and 3.1.2 and if I roll back to 3.1.1 my scripts start working again - MachineModel is subscriptable in 3.1.1, but not in 3.1.2 and 3.1.3.

Having looked at the diff on GitHub 3.1.1 - 3.1.2 I can't see why the changes would change what gets returned (apart from let it be longer). As far as I can see teh changed def receive_json(self): was returning a str and is still returning a str.

My Python is 3.7.3

wilriker

@achrn Just as an idea, have you removed all .pyc files as well as __pycache__ directories? Just to make sure nothing is stuck in half-way pre-compiled voodoo black-magic bogus.

achrn

@wilriker I ahdan't, but I was doing pip3 uninstall pydsfapi before each new install.

I've tried again - unistall, hunt for any pydsfapi traces and obliterate them (including fo ~/.local/lib/python3.7 where I found a few). I'm not sure I want to remove all __pycache__ and *.pyc - plus it would take a while to hunt them all down. There are no *dsf*.pyc or pydsfapi* (except the .gz I'm installing from) files anywhere on the machine before I do a pip3 install

After that, still the same behaviour - MachineModel object is not subscriptable.

I have a relatively simple workaround, because if I don't transfer a full model, my first 'patch' gets a whole model. So I can easily change my scripts to start with an empty dict, and use get_machine_model_patch() to obtain a 'patch' containing the whole model (as a nice predictable string). Then I apply that patch to the empty dict, and that gives me a dict of the machine model, and the script proceeds from that point fine.

wilriker

@achrn I have a vague theory of what might be the cause for this. But I can probably only test next week.

achrn

@wilriker No problem - as I say, I've got an easy workaround, so there's not a rush on my account. Thanks.

wilriker

@achrn said in pydsfapi [v3.1.3] - Official Python Client Library for DSF:

My scripts that previously did that then e.g. for h in machine_model['heat']['heaters'] are suddenly reporting TypeError: 'MachineModel' object is not subscriptable.

I see the same problem with 3.1.1. It works for me (both versions) if I use

for h in machine_model.heat['heaters']

i.e. trying to access heat as a member rather than a dict key.

achrn

@wilriker said in pydsfapi [v3.1.3] - Official Python Client Library for DSF:

It works for me (both versions) if I use
for h in machine_model.heat['heaters']

Yes, that works for me too in 3.1.3.

However, it breaks my patch-applying routine, so I think it's more work for me than my other workaround (at least, for my existing scripts).

wilriker

@achrn I don't know why this works for you in 3.1.1 (as it does not for me) but I found a simple way of getting it to work. I would though hold that off for version 3.2.0 because you said you have a working alternative solution.

achrn

@wilriker said in pydsfapi [v3.1.3] - Official Python Client Library for DSF:

I would though hold that off for version 3.2.0 because you said you have a working alternative solution.

Yes, no problem for me - I have what I wanted working now, so don't rush on my account.

wilriker

Release v.3.2.0

I am pleased to announce the release of version 3.2.0 of pydsfapi that syncs all the changes introduced with DSF 3.2.x.

This release also contains some major contributions and cleanups from @resam, thanks for that!

The package can be downloaded as usual from GitHub Releases page but will probably be available to be installed via pip in the future.