Men, though naturally sympathetic, feel so little for another, with whom they have no particular connexion, in comparison of what they feel for themselves; the misery of one, who is merely their fellow-creature, is of so little importance to them in comparison even of a small conveniency of their own
 Adam Smith, The Theory of Moral Sentiments (1759). copy citation

Context

“In order to enforce the observation of justice, therefore, Nature has implanted in the human breast that consciousness of ill desert, those terrors of merited punishment which attend upon its violation, as the great safeguards of the association of mankind, to protect the weak, to curb the violent, and to chastise the guilty. Men, though naturally sympathetic, feel so little for another, with whom they have no particular connexion, in comparison of what they feel for themselves; the misery of one, who is merely their fellow-creature, is of so little importance to them in comparison even of a small conveniency of their own; they have it so much in their power to hurt him, and may have so many temptations to do so, that if this principle did not stand up within them in his defence, and overawe them into a respect for his innocence, they would, like wild beasts, be at all times ready to fly upon him;” source