I agree, this is something I'm exploring and wrote a NIP for it I will add an "unfollow" and an "unfollow only in Olas" options and something similar for Follows.