@Phaedrux
I think I have fixed the issue.
The SBC and Duet had external PSU 5v, the cable for SBC was going in the same path where the Bed cable is. Somehow, those two were interfering with each other (inducing current/voltage) and creating issues. Also, my ribbon cable was going under the duet which wasn't helping as well.
I have disconnected 5v PSU and now the Duet powers up the SBC. The ribbon cable is above the duet. I had a successful 2.5h print.
Those parts are very sensitive to be honest.