The visitors won the shoot-out after an enthralling match in Sarajevo finished 1-1 after extra-time.
Liam Boyce scored the winning penalty for Ian Baraclough’s men after Bailey Peacock-Farrell saved from Haris Hajradinovic and Edin Visca missed.
Niall McGinn equalised for NI after Rade Krunic had opened the scoring.
NI will now play Slovakia in the play-off final in November after they defeated the Republic of Ireland on penalties in Bratislava.
Boyce had only come on as a substitute in the final minute of extra time, along with Conor Washington, who scored NI’s fourth penalty after George Saville had blazed the third one over the crossbar.
In what was the first penalty shoot-out in NI’s history, Peacock-Farrell, who had made a number of decent stops during the 90 minutes, dived to his left to deny Hajradinovic before Visca struck the crossbar.
The evening had started well for Bosnia when they opened the scoring in the 13th minute. A long ball down the right hand channel found Branimir Cipetic, who was being marshalled by Jonny Evans before the Leicester City defender slipped and let the full-back break into the box.
He cut it back to Krunic, who had stepped off Stuart Dallas to find space in the box, and the AC Milan man calmly side-footed home his second international goal despite Bailey Peacock-Farrell getting a foot to it.
McGinn’s equaliser came eight minutes into the second half when the Aberdeen winger latched on to a long ball to make the most of a ricochet and bustle his way into the box before delivering a composed finish.