And the axe comes down again opening yet another NFL head coaching job.
Colts vice chairman Bill Polian and his son, general manager Chris Polian, were already let go after the 2-14 season.
Caldwell, a soon-to-be 57-year-old, was 26-22 in his tenure with the Colts with two postseason appearances and one Super Bowl loss.
What Should Colts Do With No. 1 Pick?
His Colts were beaten up by the New York Jets in the first round of playoffs a year ago, and the trend continued this year.
The biggest difference in 2011 was that the offense couldn't help overcome the defensive shortcomings without Peyton Manning.
No. 18 is said to be recovering finally, and Colts owner Jim Irsay has made some very contradictory remarks regarding what he wants to do with the No. 1 overall pick.
That decision no longer concerns Caldwell.
The coach has worked exclusively with wide receivers and quarterbacks throughout his coaching career, and could opt for an assistant's job or possibly a return to the college game in some capacity.
Although losing Manning is no fault of his own, the inability to fix an ailing run defense or find competitive quarterback play in Manning's absence is.
He knows this and Colts beat writer Jason Spells quotes the former coach as saying:
As the former head coach, this is all he can say.
The Colts, meanwhile, will need to go forward with caution. Who they choose to replace the coordinators, Caldwell and what to do with the No. 1 pick will shape this organization for the next five to 10 years.
Expect the coaching decision to hinge heavily on what the organization wants to do at No. 1 overall.