Spurs have been linked with a move for the Chelsea striker, whose arrival at White Hart Lane would doubtless add to Defoe's frustration at his lack of opportunities this season.
But Sherwood believes it would be a mistake to let Defoe leave, and said: "It's hard enough to go and sign a player like Jermain Defoe, so you don't want to let him walk out your door readily. He obviously hasn't played as much as he would've liked but that's due to having a big squad."
Sherwood also admits that, although the Tottenham staff are big fans of Drogba, he is not convinced the Ivory Coast star's wages would be within the club's budget.
"We know what quality he possesses, Didier Drogba," he added. "Whether the finances are going to be available for us to achieve that remains to be seen. It'll be a hefty lot of money for a player of his quality."
Whether it is Drogba or someone else, Defoe appears certain to face further competition for his place up front next season, with Tottenham well known to be in the hunt for a striker.
It could be manager Harry Redknapp unearths another bargain in the Rafael van der Vaart mould.
And Sherwood, who has been scouring Europe for players to bring to White Hart Lane, said: "We're not alone when we're looking out for them, though. It's going to be a bunfight.
"There aren't any teams who can compete with the financial muscle of the Chelseas and Man Citys - and there's no shame in admitting that.
"It's very difficult to find anyone better than we've got without spending a zillion pounds."
Source: PA
Source: PA