Impersonalists undergo severe austerities and penances to merge themselves into the Brahman effulgence, or impersonal Brahman existence. But their minds are not free from material contamination; they have simply tried to negate the material ways of thinking. That does not mean that they have become liberated. Thus they fall down.
In the Bhagavad-gītā it is stated that the impersonalist has to undergo great tribulation in realizing his ultimate goal. At the beginning of Śrīmad-Bhāgavatam it is also stated that without devotional service to the Supreme Personality of Godhead, one cannot achieve liberation from the bondage of fruitive activities. The statement of Lord Kṛṣṇa is there in the Bhagavad-gītā, and in Śrīmad-Bhāgavatam the statement of the great sage Nārada is there, and here also the demigods confirm it. "Persons who have not taken to devotional service are understood to have come short of the ultimate purpose of knowledge and are not favored by Your grace." The impersonalists simply think that they are liberated, but actually they have no feeling for the Personality of Godhead.