I had my pair of Tannoy Canterbury situated in a room with similar stone design as pictured. Can't say that they sounded noticeably different than in another room that did not have stone. Stone is not optimal but it is not as bad as having glass windows or doors at the first reflection points.
You mentioned that it will not be a dedicated room so I am not sure if you intend to listen from the "sweet spot" or not"
If yes, then:
If it is possible to incorporate some aesthetically pleasing acoustic treatments in the form of art or similar, at the primary reflection points that would be visible in a mirror that is moved along the wall by someone while you are sitting in the sweet spot, that would go a long way to taming the most annoying harshness that causes fatigue.
It is not so much the reflected sound itself as the confusing information from the slight delay of the sound that has to be processed by the brain that causes fatigue. The spatial cue CPU gets overloaded.
