"Just got to be smart about the situation," Lue said, via ESPN.com's Dave McMenamin. "I know he wants to play and play a lot, but we've got to be smart because we play every other day, so there's not a lot of time really to recover."
Lue's comments come after James said he was exhausted following the Cavs' 105-101 victory over the Indiana Pacers in Game 7 on Sunday.
"I'm burnt right now," an exhausted James told reporters. "I'm not thinking about Toronto right now until tomorrow. I'm ready to go home. Can we? I'm tired. I want to go home."
James played 43 minutes in Game 7 against the Pacers. He averaged 41.1 minutes per game in the series after leading the NBA with 36.9 minutes per game during the regular season.
Lue noted that James only looks tired "after the game, but not during the game."
The Cavaliers, who have won the Eastern Conference in each of the previous three seasons, will play the Raptors in Game 1 of the Eastern Conference Semifinals on Tuesday at Air Canada Centre.