Yes, if you have that marker color selected, right-clicking on the existing marker should remove it. Or, if you have another color selected, holding down (alt, ctrl, cmd) and right-clicking should remove it no matter the color.
There really does need to be a way to remove markers.