Yes, it would be threatening to see citizens with long guns walking around in public in the same way it would be threatening to see law enforcement walking around in public with long guns. Long guns = combat in my mind. Psychologically then, for me, seeing them in public by either civilians OR law enforcement in peace time means something is wrong; something is bad.
I live in an "open carry" state. People have to go through training and obtain a permit to carry here. I've never actually seen a civilian walk around with a long gun in my life other than hunting. However, I have seen civilians "open carry" with a holstered sidearm. It's very rare to even see that here though since probably 99% of those with a "permit to carry" do so, concealed. There's an argument that I think has validity that open carrying a sidearm in public is stupid and dangerous to the person carrying it.
I will say on the rare instances where I've been in public and I've seen a civilian "open carry" a pistol, I have not felt threatened by it.