Apparently the culprit was the wiring. I wired the positive fan wires to the buck converter output (12V) and the negative fan wires to Duet fan grounds according to the documentation, specifically Mixed-voltage setups are not directly supported, but the Duet switches the fans' connections to ground, so you may be able to connect each fan's positive side to the appropriate voltage and its negative side to the appropriate pin on the Duet.
This results in an average of 8V across the fans when PWM is set to 0%.
Scrolling down I found the section Using 12V fans when VIN is 24V
and it mentions wiring the output of the buck converter to the center pin of V_FAN. This makes the voltage average go down to about 0.4V on the fan output when PWM is set to 0% and the fans behave as expected.
Also mentioned in that section is leave the VFAN pin of that connector not connected
. While I assume that this refers to the positive output for the fan it gave me the idea to remove the V_FAN jumper and this also made the fans behave as expected.
Overall this mild frustration could have been avoided by reading the docs more carefully
For those of us too eager to get results fast may I suggest adding the part about disconnecting the V_FAN jumper to the section I quoted above since that is what people first see when navigating to that page? Either that or a link to the section below it which explains in more detail how to wire mixed voltage fans.
One last thing, how do I mark a thread as solved? I can't seem to find that option anywhere