The Reds, who hold a competition-record eight League Cup titles, looked as though they were going to sail into the next round when Raheem Sterling burst through a couple of challenges to fire them into the lead. Then Daniel Sturridge doubled the advantage from a precise Steven Gerrard pass.
Brendan Rodgers then saw his side crumble in the second period.
A header from Yoann Arquin reduced the Magpies' deficit, before Adam Coombes sparked delirious scenes in the away end when he tapped home the equaliser with just six minutes left.
Liverpool lost to third-tier opposition last season when Oldham Athletic dumped them out of the FA Cup. They would have been fearing another defeat when the match entered extra-time, especially when Kolo Toure was stretchered off to join Aly Cissokho and Joe Allen on the injured list, leaving the Reds with 10 men.
However, Sturridge came to the rescue when his low shot squirmed through the grasp of goalkeeper Bartosz Bialkowski to make it 3-2 at the end of the first extra period, with substitute Jordan Henderson adding a fourth in the second half.
Here's what the Reds will have learned from an eventful night.