When the boundaries of a district which is a component of a high school district or a unified school district, as defined in Section 35515, are for any cause changed to include territory in, or to exclude from, the district, the territory added to, or removed from, the district shall, at the same time, be included in, or excluded from, the high school district or unified school district.