Speaking to TMZ Sports, Ortiz said "people can get the wrong idea" about the message the fans were trying to send.
"You don't wanna add more s--t to the s--t that is already out there," he said.
The banner was displayed on the Green Monster at Fenway Park in the middle of the fourth inning of Wednesday's game between the Red Sox and Oakland Athletics.
One anonymous member of the group told CSN New England's Evan Drellich where the motivation to display the sign came from.
"There were originally about eight people involved who had this idea, and those eight people come from various organizing groups in the Boston area," the group member said. "Mostly groups that affiliate with racial justice causes. And the banner came in response to the racist comments at the beginning of the season at Fenway [that Adam Jones spoke of]."
Security ejected four fans from the stadium for displaying the sign, with a Red Sox spokesperson telling reporters it was a "violation of the club's policy prohibiting signs of any kind to be hung or affixed to the ballpark," per ESPN.com.
Per Jen McCaffrey of MassLive.com, the banner was displayed for around a minute before security took it down.