I had never thought of this before, but I think that the fact they hadn't their wands with them was enough as to retain them there. But James did have his wand: he was making puffs for Harry. Anyway, the way the Fidelius Charm works, it wouldn't have been possible. It's such a complex charm that it's not so easy to flick in and out it.
I don't believe Apparation could be bad on babies. If there's the side-long Apparation (was it called like that?), to allow you carrying kids or people unable to Apparate by its own, that wouldn't be a problem at all.
Before I found out you need you wand to Apparate or Disapparate, I tried hard to do it. I thought that becoming relaxed and trying to mix with the universe would make the trick