yes, passive xovers are dependent upon the impedence. if the passive xover is designed for a 4ohm driver and say, for example, provides a HP of 2kHz, changing to an 8ohm driver now makes the HP xover point 1kHz. not only would this overlap with the midbass driver, but possibly be beyond the safe operating frequency response of the tweeter, leading to damage.
basically, if you're going 'custom' like that, then customise the xovers to match, with either a new passive xover or going 'active'.