@Ghazi I've tested alot lot now.
The errors I had were from random angles and trying to move there. This is not possible always, as source and targets are in different work modes. When I don't try to change work modes, it works as expected.
(explanation work modes, for users who do not know it: every position and orientation on a 6 axis robot can be reached by different robot arm positions, up to about 8 possibilities. They can be only crossed by explicit robot angle changes).
I'll proceed with creating the firmware now. I currently can work at weekends, so my goal is to finish it next weekend.
When building the CoreXY 5 axis, please be aware that the A axis has a critical angle of 0 degrees where a singularity can happen, because the C axis can be parallel to the Z axis. This should be avoided, e.g. by staying in the positive or negative A angle. Mechanical construction so that A can stay in negative or positive, I mean maximizing this range, would IMHO be the best solution. If think, if the goal of the 5 axis is to print overhangs, the maximum tilting needed would be a parameter.
The singularity can be crossed by setting C axis rotation to 0 (the firmware can do this), but the result is a movement which is not correct. I'll probably add a new flag in configuration to turn on or off the possibility to run in singularity or not. If not, outputting an error message to the console and stop printing. (to allow finding this in simulation mode)