@misterjtc I've been trying to reproduce it but so far without success with a Mini 5+, RPi 5, and v3.5.2. There is no limit for connection retries.
How do you keep the Duet powered off? If it's connected via the shipped ribbon cable, I'd expect it to be powered from the Pi anyway (5V). To keep my board "off", I wrapped a cable tie around the board which pushes the reset button continously.
Can you confirm the spidev buffer size is set to 8192 bytes? If it's too low, DSF and RRF may be unable to exchange data.
If you have a spare microSD card, please try one of the official DuetPi builds. They are based on the official RPi OS, too.
If it happens with DuetPi as well, try to log on to your Pi via SSH and check the output of journalctl -u duetcontrolserver -ef. This is what my output looked like after 13mins of the board being unavailable:
Sep 19 11:35:51 duet3 DuetControlServer[726]: [warn] Lost connection to Duet (Timeout while waiting for transfer ready pin)
Sep 19 11:48:32 duet3 DuetControlServer[726]: [info] Connection to Duet established
After that, config.g was executed as expected.