You can literally just tell them no. Gay men rarely get offended by rejection they just move onto the next guy, they may even apologize to you for thinking you were gay and hitting on you. At least in my experience.