Can you elaborate on your problem a little more? By default, hosts in the inventory that are NOT part of the inventory source will still remain in the inventory after sync.
In the inventory source page, there are “Overwrite” and “Overwrite variables” toggle buttons. If overwrite is checked, those extra hosts WILL be removed.