Lee, who became a first-time father in February, explained his reasoning to ESPN's Mike Reiss:
"This is a big sit-down process I had, with me and my significant other, as far as family goes. The risk factor in which we believe that's going out there, it just wasn't worth it in a sense. Just too many unknowns.
"We knew New England, honestly, had a great system in place. But I told Coach [Bill Belichick], at the end of the day, I wouldn't be doing him justice going out there because I wouldn't be able to give him my all knowing my family was back at the house worrying about their situation."
Lee is the seventh Pats player to opt out, joining linebacker Dont'a Hightower, safety Patrick Chung, offensive tackle Marcus Cannon, running back Brandon Bolden, fullback Danny Vitale and offensive lineman Najee Toran.
Lee signed a one-year contract with the Pats this season and was expected to compete for a roster spot with the likes of Jakobi Meyers, Gunner Olszewski and Jeff Thomas.
His contract will roll over to next season, and Lee told Reiss that he plans to play in 2021.
The 28-year-old veteran spent the first six years of his career with the Jacksonville Jaguars after they selected him in the second round of the 2014 NFL draft out of USC.
Lee's best season was in 2016, when he set career highs with 63 receptions for 851 yards and three touchdowns. He was also fairly productive the following year with 56 grabs for 702 yards and three scores.
He missed the entire 2018 season with a knee injury, however, and wasn't much of a factor in six games last season with three catches for 18 yards.
While Belichick has yet to comment on Lee's decision specifically, he did express support Friday for the other six players who had chosen to opt out: "I respect all of them. I talked to those guys and they explained their situations, and they had to make the decisions that were best for them, and I totally respect and support it."
Lee was a low-risk, high-reward signing for a Patriots team that got little production out of its wide receivers last season aside from Julian Edelman.
With one potential option out of the picture, Belichick will look to Mohamed Sanu, second-year man N'Keal Harry and free-agent signing Damiere Byrd to step up as secondary targets.