I think I have found the answer. I added these 2 lines in the start g-code and it seems that Z is adapting to the height map. I need to do testing / tuning in the coming days, but happy so far.
G29 S1 ;Load the height map from file and activate mesh bed compensation
M376 H5 ;Taper off bed compensation after Hx mm