Most unusual...
Can you please share your config.g and the results of sending M122 and M98 P"config.g" in the gcode console?
When you say it follows the extruder motor and not the driver, that makes me think it's the motor wiring either external or internal to the motor itself, however, you also say it happens with both extruder motors depending on which is being used at the time.
Can you share some photos of the extruders and the wiring? Maybe there's something odd going on.
Are you able to re-crimp the extruder motors connectors?