English Translation
It’s a pity you have to go.
English Translation
It’s a shame you have to go.
As I am not native speaker in English, I was also a bit confused with the translation.
After some googling I figured out, that “it’s a shame that” is some kind of an idiomatic expression. Nothing offensive was meant in this sentence 
However, it is better to use alternative you proposed if you’re not sure whom you are talking with(whether person is native speaker or fluent in English)