Generally, I agree with your sentiment. But what I see mentioned often is this idea that something needs to be consciously noticed or registered or discussed to have an impact.
The designed environment influences people -- literally, influences them physiologically like heart-rate, but also psychologically and emotionally -- even if they consciously don't know it's happening or why. There is a scientific discipline and literature around this.
Non-designers, non-fans, regular people, are influenced by whether things feel good and right or when they feel bad and wrong, when things are clear and meaningful or unclear and confusing, comfortable and relaxing or uncomfortable and threatening. And it can come down to the paint color, or lighting temperature, or textures, or background sounds or font choices or countless other design-choice details.
And that's why houses, stores, restaurants, hotels, parks, offices, and everything else in the designed world are designed by designers, with intention -- for better or worse. If this stuff only mattered to a small group of super-sensitive people, the designed world wouldn't be professionally designed. Why bother?
So when people claim that because people don't notice, don't discuss the design, it doesn't matter, it's silly. A portrayal of people as these zombie cave-people grunting enthusiasm about a roller coaster as if they are not also influenced by everything else around them, ridiculous.