Small vesicules continuously bud from the ER membrane and fuse into the Golgi, and this vesicule trafficking is also involved in adjusting the cytoplasmic membrane's composition and extent. This is a defining feature of the endomembrane system.
The membranes of chloroplasts and mitochondria are not involved in this vesicule trafficking, which is why they are not considered part of the endomembrane system (even though they "have membranes" and "are inside the cell", which is the etymology of the term "endomembrane").