Setting up all the DNS records right takes some effort, yes. You will need a VPS or an ISP that gives you a static IP and the ability to set the PTR record for it. Other than that, self hosting without losing email is perfectly possible. I have been doing it for the last 10 years or so.