A father is apt to be less attached to a child, who, by some accident, has been separated from him in its infancy, and who does not return to him till it is grown up to manhood.
 Adam Smith, The Theory of Moral Sentiments (1759). copy citation

Context

“Though in a particular instance, the circumstances which usually produce those natural affections, as they are called, may, by some accident, not have taken place, yet respect for the general rule will frequently, in some measure, supply their place, and produce something which, though not altogether the same, may bear, however, a very considerable resemblance to those affections. A father is apt to be less attached to a child, who, by some accident, has been separated from him in its infancy, and who does not return to him till it is grown up to manhood. The father is apt to feel less paternal tenderness for the child; the child, less filial reverence for the father. Brothers and sisters, when they have been educated in distant countries, are apt to feel a similar diminution of affection.” source