Yup I've had this problem as well. I think consistency and generous use of white space helps to alleviate this. As well as eliminating non-clickable items as much as possible. Ideally the content itself would be actionable without the need for buttons everywhere.

Realistically if something isn't actionable on mobile you should really be questioning what it is doing there, or whether you can simplify the interface by making it interactive.