Other billionaires and financial experts also say investing in a family member’s new business can lead to complex trouble and strained relationships.