Levi doesn't read people very well, but even he can see split second of hurt there. "I-its not you! I've done that to multiple people here. I just, uh, I-i never really got touched much at home. A-at least not in a friendly way."

Just the reaction of an abuse victim reflexively fearing more. Especially during his years in the orphanage physical contact was far from pleasant.