women's quarters in the traditional house of a Muslim family

Harem means the women and the place where they live in a polygynous household. Most men are not allowed in there. The word harem comes from the Turkish word haram, meaning a sacred, forbidden place. Harems were meant for the royal upbringing of the future wives of noble and royal men. These women would be educated so that they were ready to appear in public as a royal wife.

Pool in a Harem by Jean-Léon Gérôme, about 1876
Newly Arrived (Giulio Rosati)

In the West people thought that harems (especially those in the Ottoman Empire) were brothels full of women who gave themselves up to please rich men.

In modern terms the word maybe used to describe a group of admiring women attracted to one man.