When you say "It made no difference to me"—I think you refer to how you treat your uncle. You're making a decision not to treat him as an inferior, as others might have done.

But there was in fact a difference, and one that greatly formed his life.

You made a decision to ignore the "difference" as a social strategy, but that causes problems if trying to reflect on his life, or by extension, the lives of people like him.