The Senegal international, who is out in Tunisia for the African Nations Cup finals which begin this weekend, is not happy at failing to secure a regular place with the Serie A side.
"It's a crazy club with lots of crazy things happening there," Coly told the Press Association..
"One week I'm in the first-team playing and another week I'm out without any concrete reason. I'm fed up with the situation.
Coly moved to Perugia in the summer after an unsuccessful six month loan spell at St. Andrews in which he managed just a handful of appearances and was not offered a permanent deal.
"I didn't have a fantastic time in England either," he added.
"I had hardly arrived when I had a toe infection and stayed in the hospital for 10 days."