pip won't install dsf-python-3.5.1rc1

achrn

@Falcounet I'm trying to test dsf-python-3.5.1rc1 but if I do sudo pip3 install --pre dsf-python the Pi responds

ian@delta:~ $ sudo pip3 install --pre dsf-python
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

This is on a completely clean brand-new Bookwork DuetPi. The only thing I did was set a hostname, set username and password, and enable ssh in raspberry imager (if I don't do that, I can't connect - user pi password raspberry and all obvious variants I can think of give me permission denied).

This was 64bit lite image, on a MB6HC, with Pi4.

If I copy the source across and sudo python3 setup.py install it seems to install OK, but the http endpoint example doesn't work right - but I think best to keep one thing per thread. Is it worth continuing to test with having done the install this way or do I really need to get a pip install working for further testing to be worthwhile?

Falcounet

@achrn I believe it's ok to install dsf-python system-wide.
If you don't want to, another option might be to install it for the user which will run your scripts or in a virtual environment.

achrn

@Falcounet I'm entirely happy to have dsf-python system-wide, but I think pip is saying it won't install dsf-python and saying I should use apt, but I don't think there's an apt package for dsf-python - apt-cache search dsf-python doesn't find any likely contenders.

If I put dsf-python into a virtual environment I don't think that the DSF running on the system will find it or use it - it's not in that virtual environment. Will the DSF plugins system find and use a dsf-python in a virtual environment?

Having done some digging, it seems pip has been deliberately disabled:

https://www.raspberrypi.com/documentation/computers/os.html#about-python-virtual-environments says "In previous versions of the operating system, it was possible to install libraries directly, system-wide, using the package installer for Python, commonly known as pip. ... In newer versions of Raspberry Pi OS, and other operating systems, this is disallowed. ... from Bookworm onwards, packages installed via pip must be installed into a Python virtual environment using venv. A virtual environment is a container where you can safely install third-party modules so they won’t interfere with, or break, your system Python." which suggests that install instructions using pip aren't going to work on newest DuetPi (which is Bookworm) or any future version.

So having been through a time where setup.py was deprecated and pip was advocated, pip is apparently now not usable, forcing users back to setup.py (unless you're happy to have your python running only is an isolated sandbox). I think. (I despised python even before this...) That or dsf-python is going to need an apt package in the Duet3D repository.

resam

@achrn not sure such a hard stance is needed here. Its perfectly fine to override this behaviour if you know what you are doing. The pip output you posted contains all instructions you need:

You can override this, at the risk of breaking your Python installation or OS,
by passing --break-system-packages.

Not pretty, I agree, but certainly an easy fix to get the package installed "the old way".

Falcounet

@achrn said in pip won't install dsf-python-3.5.1rc1:

@Falcounet I'm entirely happy to have dsf-python system-wide, but I think pip is saying it won't install dsf-python and saying I should use apt, but I don't think there's an apt package for dsf-python - apt-cache search dsf-python doesn't find any likely contenders.

Yes because we don't maintain a deb package for dsf-python to be installed with APT. pip does it just fine.
You are right about pip refusing to installing it system-wide though, it needs an override to work.
But as said earlier, you can still install it using pip for any user (so without sudo) or in a virtual environnement.

If I put dsf-python into a virtual environment I don't think that the DSF running on the system will find it or use it - it's not in that virtual environment. Will the DSF plugins system find and use a dsf-python in a virtual environment?

Yes it will work. The virtual environnement is for python only. It will still connect to DSF using the IPC API.
For dsf plugins to work, you need to have dsf-python installed for dsf user.

Having done some digging, it seems pip has been deliberately disabled:

https://www.raspberrypi.com/documentation/computers/os.html#about-python-virtual-environments says "In previous versions of the operating system, it was possible to install libraries directly, system-wide, using the package installer for Python, commonly known as pip. ... In newer versions of Raspberry Pi OS, and other operating systems, this is disallowed. ... from Bookworm onwards, packages installed via pip must be installed into a Python virtual environment using venv. A virtual environment is a container where you can safely install third-party modules so they won’t interfere with, or break, your system Python." which suggests that install instructions using pip aren't going to work on newest DuetPi (which is Bookworm) or any future version.

So having been through a time where setup.py was deprecated and pip was advocated, pip is apparently now not usable, forcing users back to setup.py (unless you're happy to have your python running only is an isolated sandbox). I think. (I despised python even before this...) That or dsf-python is going to need an apt package in the Duet3D repository.

pip is disabled to install system-wide packages. I don't think it is disabled for a normal user.

achrn

@resam said in pip won't install dsf-python-3.5.1rc1:

@achrn not sure such a hard stance is needed here. Its perfectly fine to override this behaviour if you know what you are doing.

It's the 'if you know what you are doing' part that's the problem. I don't know anything about this change to pip or the virtual environment system, and the which of the four different ways that message might be telling me to proceed is not obvious to me.

I'm not trying to be hardline, I'm just trying to get the software installed.

@Falcounet said in pip won't install dsf-python-3.5.1rc1:

Yes it will work. The virtual environment is for python only. It will still connect to DSF using the IPC API.
For dsf plugins to work, you need to have dsf-python installed for dsf user.

I'm unclear whether installing it for (as?) a user and installing it in a virtual environment are two different options, but notwithstanding that:

I'm not sure how to install as dsf, given that the account is nologin. There is no password for dsf. sudo su dsf gives me This account is currently not available. Should I just set dsf to allow login and then login as that user and do the pip3 command? I'm a little nervous that doing that might upset something else that uses the dsf user.

If it's in an environment, I note the web page linked says "activate that environment before running any of your Python code", so that presumably requires DSF to do that step if it's a plugin? Is there a need to do anything to tell DSF of the virtual environment (or indeed which virtual environment) to run a plugin in?

Given these questions, I think I'll try the --break-system-packages option first,

achrn

@achrn said in pip won't install dsf-python-3.5.1rc1:

Given these questions, I think I'll try the --break-system-packages option first,

Thanks @resam and @Falcounet, sudo pip3 install --break-system-packages --pre dsf-python works fine (though it moans some more about how I should be using a virtual environment), and I get a working dsf-python (the send_simple_code.py example is working). I haven't tried a plugin yet, but I need to tidy up my install first - I have somehow acquired many copies of dsf-python installed at various revisions.

T3P3Tony

@achrn said in pip won't install dsf-python-3.5.1rc1:

I'm not sure how to install as dsf, given that the account is nologin. There is no password for dsf. sudo su dsf gives me This account is currently not available. Should I just set dsf to allow login and then login as that user and do the pip3 command? I'm a little nervous that doing that might upset something else that uses the dsf use

@Falcounet it would be good if we could document this given how the new version is being a pain.