The voice calling humans back to Earth was deafening, but with the planet owned by just a handful of families, visiting was expensive