Appearing on ESPN's Adam Schefter's Know Them From Adam podcast (h/t Rotoworld), Olsen said it was "too early" to say if he would hold out.
He is currently seventh among tight ends in average annual salary and 12th in guaranteed money.
According to Spotrac, Olsen has two years remaining on his contract with an annual base salary of $6.5 million.
The 32-year-old veteran is coming off his third consecutive 1,000-yard season, as he registered 80 receptions for 1,073 yards and three touchdowns in 2016.
Olsen is a three-time Pro Bowler and has firmly established himself in recent years as quarterback Cam Newton's go-to target.