“ But, to the man who under-rates himself, unless we have both more discernment and more generosity than belong to the greater part of men, we seldom fail to do, at least, all the injustice which he does to himself, and frequently a great deal more. ”
 Adam Smith, The Theory of Moral Sentiments (1759). copy citation

add

“Though our dislike to the proud and the vain often disposes us to rank them rather below than above their proper station, yet, unless we are provoked by some particular and personal impertinence, we very seldom venture to use them ill. In common cases, we endeavour, for our own ease, rather to acquiesce, and, as well as we can, to accommodate ourselves to their folly. But, to the man who under-rates himself, unless we have both more discernment and more generosity than belong to the greater part of men, we seldom fail to do, at least, all the injustice which he does to himself, and frequently a great deal more. He is not only more unhappy in his own feelings than either the proud or the vain, but he is much more liable to every sort of ill-usage from other people. In almost all cases, it is better to be a little too proud, than, in any respect, too humble; and, in the sentiment of self-estimation, some degree of excess seems, both to the person himself and to the impartial spectator, to be less disagreeable than any degree of defect.
” source