Saying it in a casual passing manner means nothing. It’s just like any other word, easily abused or used as a lie, and more often than not, there’s no true feeling behind it. It’s said in moments of infatuation and desire, not when it’s supposed to be used.
To truly call someone beautiful you have to be able to stop and truly see them. You need to be able to see inside them and see their quirks and mistakes and know that they are truly beautiful inside and out.
When someone stops and truly see you, then calls you beautiful, that’s when you know they mean it. And that is when it should matter.