In Śrīmad-Bhāgavatam (SB 3.31.33-35) it is said that one should give up all association with a person who is a playmate for women, for by associating with such an unholy person one becomes bereft of all good qualities, such as truthfulness, cleanliness, mercy, gravity, intelligence, shyness, beauty, fame, forgiveness, control of the mind and senses and all the opulences that are automatically obtained by a devotee. A man is never so degraded as when he associates with persons who are too much attached to women.
In this regard, Lord Caitanya also quotes a verse from Kātyāyana-saṁhitā: "One should rather tolerate the miseries of being locked in a cage filled with fire than associate with those who are not devotees of the Lord."One is also advised not to even look at the faces of persons who are irreligious or who are devoid of devotion to the Supreme Lord.