Debo haber entendido mal.

English Translation

I must have misunderstood.

¿Por qué no es ‘Debo haber malentendido’?

I don’t think your suggestion is wrong, it’s just another way of saying the same thing. I can imagine, though, that malentender is most frequently used in the infinitive and the split forms otherwise, depending on which is more convenient.