The concern which we take in the fortune and happiness of individuals does not, in common cases, arise from that which we take in the fortune and happiness of society.
 Adam Smith, The Theory of Moral Sentiments (1759). copy citation

Context

“But few men have reflected upon the necessity of justice to the existence of society, how obvious soever that necessity may appear to be. That it is not a regard to the preservation of society, which originally interests us in the punishment of crimes committed against individuals, may be demonstrated by many obvious considerations. The concern which we take in the fortune and happiness of individuals does not, in common cases, arise from that which we take in the fortune and happiness of society. We are no more concerned for the destruction or loss of a single man, because this man is a member or part of society, and because we should be concerned for the destruction of society, than we are concerned for the loss of a single guinea, because this guinea is a part of a thousand guineas, and because we should be concerned for the loss of the whole sum.” source