There are a lot of users using PanelDue, so if this was a firmware issue then I would expect to have had a lot more reports of it. It might just be chance that the problems went away when you disconnected the PanelDue. Or it could be that your TFT panel is faulty, for example if the backlight inverter occasionally draws a high current pulse from the 5V supply then that could compromise power integrity.
Also please check that the S1 parameter is present in your M575 P1 command, or that you have no M575 P1 command at all in config.g. The S1 parameter enables checksums, to guard against corruption of the data sent by PanelDue to the Duet.