I feel there are 2 needs in your message : having people/noise around you ("life" :-)) and also being able to interact more often/differently with your partners.
Facing that issue with "work@home" in my company, the current "solution" is usually the co-working places : you get "noise", people, and good conditions to work (might be missing at home). And because it's a melting-pot of different businesses, can creates some opportunities too !
Personally I can "live" working at distance with my colleagues, but cannot stand being alone for long.
I don't have issue working with my colleague on skype, and meet them from time to time (your retreats)
In my humble opinion : You wonder if it's possible to work remotely : Yes, alone : Not for long (but does not have to be your partners).
But that's only my personal view.
Hope it helps :-)