When we withhold from another, we withhold from God (love) within ourselves.
First we withhold affection that would connect two to make one, and will ourselves to be separate, against our nature. Painful.
Withholding is a form of denying another and creating an other.
At some point, sooner or later, we will be called to correct the distortion.