Rivers said his team was "soft" defensively in the first of two outings in Milwaukee.
"It doesn't matter who's in or out, we've got to defend better," he told reporters.
Philadelphia dropped its third consecutive game in the absence of Ben Simmons, though Tobias Harris—who missed three games with right knee soreness—made his return in Thursday's loss. Joel Embiid, who acknowledged that he wasn't fully healthy in Wednesday's loss to the Phoenix Suns, noted his role pregame and said he knew he needed to play.
But it was a struggle for the 76ers to put all of their pieces together against the Bucks, who were also riding a two-game losing streak and had lost five consecutive home games entering Thursday.
Milwaukee ended the night shooting 55.6 percent from the field and hit a whopping 20 of 40 attempts from deep. They also had the advantage on the glass, outpacing Philadelphia on rebounds 45-33.
Rivers also explained his decision to keep Embiid off of Giannis Antetokounmpo throughout the game, noting that he'd rather save the matchup for the postseason rather than the second game of a back-to-back.
Antetokounmpo went off for 27 points and 16 rebounds in the win, while Khris Middleton (24 points) and Bobby Portis (23 points) were right there with him. Jrue Holiday also logged a double-double for the Bucks with 12 points and 11 rebounds.
The 76ers fell short despite 24 points from Embiid and 20 from Shake Milton off the bench.
The two teams meet again on Saturday at 3:30 p.m. ET.