The Prime Minister has even entered the row about the use of the term after the Football Association warned supporters that use of such words could result in either a banning order or even criminal charges.
David Cameron said Spurs fans - who use the term as an act of defiance after anti-Jewish abuse in the past from rival supporters - should not face prosecution.
But Carlisle, chairman of the Professional Footballers' Association (PFA), said he agreed with the comedian David Baddiel who argued that there would be an outcry if a team with old roots in the black community appropriate racist language.
Carlisle told Press Association Sport: "Do they have a right to appropriate that term when it would be indescribably offensive to anyone else?
"David Baddiel says that's how it feels as a Jewish man going to Tottenham and hearing them chant that. If it is highly offensive to him then I think Spurs have to take that on board, because he will not be the only person.
"It is not for them to appropriate a derogatory offensive term that was used to belittle a whole section of society in a terrible era."
Carlisle, speaking at a book signing of his autobiography 'You Don't Know Me But... A Footballer's Life', pointed out that such chants breached the law.
He added: "I don't feel it's right. Spurs fans may not intend for it to be offensive but it will be perceived to be offensive by a section of the community and the law states that's not allowed - it's not even my personal opinion, that's what the law states."
Tottenham responded to the FA warning by announcing they would send a questionnaire to all season ticket holders asking if the practice should stop.
Source: PA
Source: PA