I think it's true, actually, that men as a category "don't care about lesbians," but anyone who translates that into "lesbians aren't harassed by men" is suffering from typical mind fallacy and an inability to consider the individual. Most men are thankfully fine and normal. Some men — enough men to put thorns in our sides — take a polite "no thanks, I'm lesbian" as a challenge to wear down into a yes. This is half because they've known women who called themselves lesbian but weren't (fuck them), and half because they have a hard time taking any woman's expression of her own desire seriously.
Perhaps you wouldn't do that. (Or perhaps you would to a woman you liked — not a dastardly buzzcut-and-overalls Lesbian, you know, just a woman.) But there are billions of men out there, and not all of them hew to your individual desires and codes of behavior.