So then people wear them with t-shirts and jeans, only to realize later that that was the worst fashion decision they could have made. It's like wearing dress shoes with sweat pants, but less subtle. Rock a nice fedora next time you wear a corresponding suit, and you're golden. Otherwise, you're that guy who wears fedoras and thinks he looks cool.
This probably isn't what you mean, but when you say that I think of two things:
1) You collect fedoras and don't wear them because they are a very non-essential piece of clothing aka fashion accessory 2) You wear fedoras to in memory of your grandmother who you promised on her deathbed you would wear fedoras like she always wanted you to.
While those are some interesting takes on my statement, you're right, that's not what I meant. What I meant was simply that I find the shape of a fedora interesting and I like the feel of one on my head.