Stan Lazaridis scored the decisive penalty to book Blues' place in the quarter-finals after Marvin Elliott's last-gasp extra time goal ensured the tie went the distance.
Julian Gray and Emile Heskey twice put Blues ahead but Alan Dunne and Elliott gave the home fans hope of causing an upset for Peter de Savary's first match as Millwall chairman.
The ill-feeling between the two sets of fans, stemming from the riots outside the Den following the play-off semi-finals in May 2002, was obvious from the outset.
And in the eighth minute of the match the police were called into action to stop a group of home fans from getting near the away stand.
Three minutes later Gray opened the scoring for Blues as he latched on to Heskey's pass to fire past Lenny Pidgeley.
Jermaine Wright should have equalised for Millwall on 25 minutes after Dunne had taken advantage of a mistake by Martin Taylor, but the winger scuffed his shot across the face of goal.
On loan keeper Pidgeley made two excellent close-range saves in quick succession in the 32nd minute, first keeping out Heskey's header then denying Matthew Upson from the resulting corner.
Then Dunne levelled on 57 minutes after beating Walter Pandiani near the edge of the area before curling in a shot from a tight angle which went in off the crossbar with Nico Vaesen well beaten.
Heskey had two chances to win the game in the second half. Firstly he fired wide from 20 yards on 61 minutes, then saw another shot go the wrong side of the far post with an empty goal begging.
But Millwall looked determined to win in normal time and Ben May should have done better with his 80th minute volley which flew wide of the near post.
He then set up Elliott for a long-range shot which whistled past the post with Vaesen looking on helplessly.
Heskey made up for his earlier misses five minutes into extra time when he tapped home after a mistake by Pidgeley from a Pennant corner but Elliott sent the tie to penalties with three minutes left with a thunderous strike from 25 yards that sent the home fans into delirium.
And so it went into sudden death from the penalty spot.
Nicky Butt, David Dunn and Jiri Jarosik all scored from the spot for Blues but Jermaine Pennant failed to win the tie after Barry Hayles and Ben May had missed penalties for Millwall.
However, Lazaridis cooly slotted home his penalty to send Blues into the last eight.