Haridasa Thakura, although certainly competent and qualified to enter the temple, did not want even to go near it. This is called Vaisnava humility
Haridāsa Ṭhākura replied, "I cannot go near the temple because I am a low-caste, abominable person. I have no authority to go there."
Although Haridāsa Ṭhākura was such a highly exalted Vaiṣṇava that he was addressed as Haridāsa Gosvāmī, he still did not like to disturb the common sense of the general populace. Haridāsa Ṭhākura was so exalted that he was addressed as ṭhākura and gosāñi, and these titles are offered to the most advanced Vaiṣṇavas. The spiritual master is generally called gosāñi, and ṭhākura is used to address the paramahaṁsas, those in the topmost rank of spirituality. Nonetheless, Haridāsa Ṭhākura did not want to go near the temple, although he was called there by Śrī Caitanya Mahāprabhu Himself. The Jagannātha temple still accepts only those Hindus who are in the varṇāśrama order. Other castes, especially those who are not Hindu, are not allowed to enter the temple. This is a long-standing regulation, and thus Haridāsa Ṭhākura, although certainly competent and qualified to enter the temple, did not want even to go near it. This is called Vaiṣṇava humility.