Mason Foster's interception return was wasted by inconsistent offense and a missed field goal.
For the second week in a row, the Buccaneers gave up a late lead and were left wondering "what if?" after a string of missed opportunities and egregious mistakes.
But rather than Week 1, when it was the surprising New York Jets taking the victory, it was division-rival New Orleans, who take the driver's seat in the NFC South after the win.
So what did we learn about the Buccaneers following their Week 2 loss?
This was a fairly common sight on Sunday, which is encouraging for the Bucs.
Last season, the Buccaneers finished near the bottom of the league with only 27 quarterback sacks as a defense. So it was surprising when leading sacker Michael Bennett was allowed to walk during the offseason.
The Buccaneers have shown during Weeks 1 and 2 that they don't miss Bennett in the slightest.
The Bucs brought down Drew Brees four times on Sunday, with linemen, linebackers, and even safety Mark Barron getting in on the quarterback pressuring.
Gerald McCoy was dominant from his defensive tackle position, and defensive end Adrian Clayborn was used in some creative ways to make sure he was coming after Drew Brees on a regular basis. Clayborn was even used as an outside linebacker in a 3-4 look on defense, something rarely seen from the Bucs.
But it's the play of Lavonte David that stands out. The young linebacker got to Brees twice, once on a blitz where he dominated Darren Sproles who tried to pick him up, and the other where he met a blitzing Mark Barron at the passer.
If David can continue to add sacks to his already impressive abilities as a run-stuffer, he's going to challenge Luke Kuechly for the title of best young linebacker in the NFL.
Mason Foster running back an interception that would give the Buccaneers a late lead.
Mason Foster and Dekoda Watson both intercepted Drew Brees on Sunday, and the defense held strong on a goal-line stand late in the first half to keep the score close.
In other words, the Buccaneers once again have a defense to be proud of.
Watson and Foster both snagged their interceptions as a result of dropping into coverage after showing a blitz look to Brees. For Foster, it couldn't have come at a better time.
Foster's slow, rumbling return of his interception in the fourth quarter gave the Buccaneers momentum and a lead that they should have been able to hold. He also contributed to the run defense, playing a key role in the goal-line stand in the first half.
The Tampa Bay defense was absolutely fantastic on Sunday. There's really no other way to put it.
Jimmy Graham had plenty to celebrate on Sunday.
The performance by Tampa Bay's defense was not flawless, however.
Ahmad Black and Dashon Goldson were both flagged for very obvious personal-foul infractions, extending New Orleans drives and otherwise taking away from a fantastic defensive effort.
Black's hit in particular was unnecessary and will merit a large fine from the NFL offices. He targeted the head of a defenseless receiver after the ball had already been dropped. It was the sort of play that could have caused serious injury, and the NFL is clearly cracking down in this area.
The Buccaneers also allowed the two biggest plays of the game on mental mistakes and coverage lapses.
The first was a long touchdown to Jimmy Graham, the only New Orleans touchdown of the game. The defense was trying to change into a Cover 2 defense late in the play clock, and safety Mark Barron wasn't on the same page as the linebackers making the audible.
That left open space for Graham, who hauled in a long pass and waltzed into the end zone for six points.
The other mistake was on the penultimate play of the game, when inexperienced cornerback Leonard Johnson was lined up against Marques Colston, allowing a long pass that set up the Saints for an easy game-winning field goal.
The Buccaneers should have been better aligned on defense to prevent an average-at-best cornerback from lining up against the best wide receiver for the Saints.
Schiano does not prepare his team well enough.
Head coach Greg Schiano proved on Sunday that he's not a part of the solution for correcting the mental errors of his football team.
If anything, he's a part of the problem.
After the awful personal-foul penalty on Ahmad Black, who targeted Jimmy Graham's head with a vicious hit, Schiano was busy on the sideline shouting and pitching a fit.
Was it at his irresponsible player? No.
It was at the officials.
Schiano was upset that Graham got to stay on the field after being injured by the jarring, unnecessary hit from Black. The backup safety was allowed to stay on the field for the Buccaneers, rather than being removed and scolded for his ridiculous action.
The Buccaneers also committed multiple dumb penalties that took away big plays and opportunities. The offense had multiple illegal-formation flags thrown against them, which is simply unacceptable.
The team even struggled to get the correct personnel on the field, causing the Bucs to burn timeouts that could have been important in such a close game.
Josh Freeman and the offense were not good enough on Sunday.
The Buccaneers were not good enough on offense. There's really no other way to say it.
Josh Freeman was not accurate enough and did not make the plays necessary to win. And even when he did have a brief moment of brilliance, it was ruined by a procedure penalty.
Doug Martin finally got going, gashing the New Orleans defense behind the blocking of Demar Dotson and Davin Joseph. But when the Bucs got into long-yardage situations, they most often crumbled.
And that was their downfall.
The Tampa Bay offense put its defense in an awful situation at the end of the game, and the defense did everything it could to hang in there. The offense didn't deliver the points needed to win nor did it do the defense any favors by setting them up to fail on the final drive of the game.
Don't get too used to this sight. Freeman won't be the QB in Tampa for much longer.
Josh Freeman's contract is up at the end of the season, and that means 2013 is an extremely important season for the young quarterback.
Not that you could tell by the way he plays.
Freeman was an embarrassing 9-of-22 passing on Sunday, throwing for only 125 yards with one touchdown. He also had two turnovers.
A completion percentage in the low 40s and a pair of turnovers doomed the Tampa Bay offense on Sunday, as they couldn't string together plays and move the ball into scoring range.
The Bucs did not allow Rian Lindell to attempt a field goal until the final moments of the game, proving just how ineffective they were at getting the ball across the 50-yard line and into the red zone, let alone scoring points.
Freeman was the crux of the issues on offense, as his inconsistent decision-making, poor accuracy and inability to handle pressure led to a disappointing performance that left more questions than answers about the quarterback's future.
Erik Lorig led the team onto the field with the American flag, and he led Doug Martin with great blocking.
The return of Erik Lorig was huge for running back Doug Martin.
Martin struggled to get anything done during Week 1 against the Jets, but during Week 2, he flourished again with his lead blocker back in action.
Lorig missed Week 1 with an injury but returned just in time to set the table for a 144-yard effort from Martin.
The Buccaneers will have to run to ball to be effective on offense this season, and that means Lorig must remain healthy and effective as he leads the way for Martin.
Davin Joseph lived up to his Pro Bowl potential on Sunday.
It's not easy for a guard to catch the eye of viewers during an NFL game, but that's what Davin Joseph did on Sunday against the Saints.
The Pro Bowl guard paved the way on the right side of the offensive line, setting up Doug Martin with great room to run all day long. Joseph combined with right tackle Demar Dotson to give Martin a chance to break out and have his best performance of the young 2013 NFL season.
Joseph was disappointing in Week 1, as he was playing in his first-regular season game since a nasty injury sidelined him during the 2012 preseason. But it's clear it didn't take long for Joseph to regain his form, as he was dominant on Sunday against New Orleans.
The Buccaneers will need continued strong performances from Joseph and his fellow linemen if they are to have offensive success this season, because it's clear that quarterback Josh Freeman will not be able to consistently move the Buccaneers through the air.
Rian Lindell only got one field-goal attempt on Sunday, and it ended in disappointment.
This offseason, Tampa Bay kicker Connor Barth was ruled out for the season after injuring his leg at a charity basketball game, as reported at the time by Ian Rapoport of NFL.com.
The Buccaneers wound up bringing in Rian Lindell to be the placekicker, and he failed to deliver in a key situation on Sunday, as he missed a 47-yard field goal that set the Saints up with good field position for an eventual game-winning drive.
Lindell doesn't have the same range or accuracy as Barth, nor is he used to the climate and field conditions at Raymond James Stadium. Lindell did nail a clutch field goal against the Jets in Week 1, but that was in the more familiar New York climate.
The field in Tampa was wet, and there was a bit of swirling wind. Lindell wasn't able to handle the circumstances, and the Buccaneers were unable to seal a victory as a result.