I definitely think that when he was a baby, they had to have taken proper care of him. They also must have given him at least
some interpersonal, real, human contact, or he would have grown up a lot different. Maybe even unable to talk or interact with other people.

That's what children do if they don't get any affection when they're babies.
I also wonder about the abuse. Definitely emotional, almost definitely physical...and I question about whether the neighbors would have actually noticed. I mean, there have been enormous atrocities committed against children all around the world, where the neighbors
should have seen, they
should have noticed, and they inexplicably...didn't. So the Dursleys could have abused him all they wanted, in all likelihood, and the neighbors probably would have been blind to it all...even if willfully blind. After all, you don't want to admit that your neighbors are child abusers, now do you? Regardless of what's really happening.
I do think that at least Petunia cared about Harry, though. Even if she did hate Lily for being a witch, Harry was her link to her sister...I mean, he was Lily's son! So he was a connection back to her sister. And ya know, even if you really dislike your family, blood is blood. There's a reason everyone knows the saying "Blood is thicker than water."

It also seemed like, in general, most of the fear Harry exhibited was because of Vernon, not Petunia, which would fit with her really caring about him, even though she hid it.