I'll worry about it once there's peer-reviewed studies that prove it.

Is it much better to have wired headphones, but the phone directly in the pocket?

You can't feel the difference? Headaches, subtle noise, feeling the magnetic field? Much less with wired headphones. We all have great instruments that tells us what is good for us and what is not. Many things are tolerable, but only up to a certain amount. Go out on nature for a week if you need re-calibration then contemplate and feel how the devices around you effect you for the better or worse.

I feel the direct contact with the phone much more...