Recently I read a posting from an audiophile who was complaining that his new (and supposedly better) DAC necessitated turning up the volume on the quiet passages and down on the louder ones. In the pro world that's referred to as "gain riding" and in the bad old days before automated boards and Sonic Solutions, engineers were often forced to resort to this when their gear didn't have the ability to capture the entire dynamic range of a live or studio performance.
But when a listener feels the need to gain-ride a commercial recording it's usually because the dynamic range of the recording exceeds their system's ability to deliver dynamics in a linear manner. The rest of their system could even be "too good" for their room or speakers - causing distortion by exceeding the speaker's or room's dynamic range. And yes, you can overdrive a room just as you can overdrive speakers.
Another, increasingly common problem is the listening environment might be too noisy to support a dynamic recording...
My own live classical orchestra recordings routinely have a dynamic range, from loudest to softest, of over 40 dB. That is loudest to softest music, not signal. The S/N range of all recorded sound on these recordings is closer to 60 dB. In an especially noisy environment, such as inside a moving car or on a subway, there's no way a system will successfully reproduce that entire range. Either you're going to bury the quiet passages under ambient walla or the high levels of the loud parts are going to hurt both you and your stereo. The only solutions are manual gain-riding or some automatic dynamic compression, so the dynamic range between loud and soft will fit inside the dynamic range capabilities of the reproduction system.
Some audiophiles will claim there's no such thing as too much dynamic contrast and better wider dynamic swings are ALWAYS a good thing, but this may not be the case. Ideally, for optimum fidelity and linearity a system's dynamic range shouldn't need any adjustment or compression, but in the real world the listening environment's ability to handle wide dynamic swings must be considered along with the gear.
There are some listening environments, such as a subway, where the only way to hear the full dynamic range will be through headphones, and only if the headphones offer sufficient noise isolation. Unfortunately, it's not as easy to noise isolate an entire room (but doable.)
An inconvenient fact about wideband dynamic contrast is that not all systems can handle ALL of the truth. And sometimes no amount of reference gear will help - the room or the environment itself is simply too noisy and incapable of supporting wide dynamics. Sometime the best option (and least expensive solution) is to listen somewhere else...