Well if you go to high, you will get spotted due to posture and gait, and look of the shoe. Shoe lifts are no longer uncommon, they are mainstream now (you see them all over being sold on social media.
But to your question, yes if you go to high, people will notice.
As for myself, i was noticed a few times because I would enter a home where shoes were not permitted, or visited a resort with a lovely woman, and obviously had to get in the water. The more you act like you care, the more they will care, if you can get to a place where it does not change you at all, you will be fine.
3 inches for events (cool)
2 inches daily lifestyle stuff
1 to 1.5 inch sandle for beaches etc (not noticeable).
And finally, when you cant have any, fk it man - you did your work, you got what you got, just do not care about it. Look at it like a woman would, if she had to swim in a pool, and could not wear her push up pads... no one cares but her. This is really what it is like.