Let go of me.
Isn’t it more like “let me go” than “let go of me”?
If I could reply in 4 characters I’d say ‘Yes.’.