Johnson broke the deadlock with a header on 57 minutes before Madine struck on the counter in stoppage time.
It was the hosts who made a fast early start, former City man Mark Marshall putting a low ball in the area in the first minute, but Reda Johnson did well to cut out the cross.
Vale had the first shot after four minutes. A well worked corner routine gave Frederic Veseli a sight of goal but he shot just wide.
City rode out the fast start from the Valiants and began to play some passing football. Josh McQuoid and Jim O'Brien saw some early touches, looking to make the most of a vast Vale Park pitch.
The Sky Blues had their first sight if goal in the 11th minute. Some smart play saw McQuoid fly down the wing, he slid it to Barton whose shot was defected beyond the far corner.
It was an open first quarter of an hour and City were really starting to find their groove, seeing a lot more of the ball than the hosts. However Vale had a big penalty shout turned down after 17 minutes when Marshall nipped past Johnson and was brought down, but referee Mark Heywood was unmoved.
Neither keeper was tested in the first half an hour, however Gary Madine was inches away from putting the Sky Blues one goal up when he looked to latch onto a centre from Johnson, but Veseli just stole the ball away at the last minute.
In the 31st minute Vale had their best chance yet. An in-swinging corner was met by Chris Robertson, but Ryan Allsop got down low to save well and the ball was cleared.
It was an even game, with both teams settling down, not wanting to be the first to flinch. However a defensive mix up almost gave City the lead three minutes before the break when Chris Neal and Robertson both went for a Jim O'Brien cross, but Madine couldn't capitalise.
That was the last action of a stop-start first half, but City had every chance of coming away from Vale Park with points in the bag.
The second half was started brightly by City, as they looked to take the game to Vale early on. Madine had a shot four minutes after the break which travelled wide of Neal's right hand post.
Seven minutes into the second half and Vale had a big chance to break the deadlock. Marshall performed a give and go and broke into the penalty area, but Allsop did brilliantly to spread himself and dive at his feet, grasping the ball.
It was great early pressure from the Sky Blues though and it paid off when they broke the deadlock just before the hour. A great cross from Matthew Pennington was high and looping and Johnson rose highest to power his header in the far corner.
It was Johnson's sixth goal of the season, taking him top of the Sky Blue goalscorer charts. It gave City a massive lift and the travelling Sky Blue Army were now in full voice.
City understandably started to play on the counter as Vale desperately looked for the equaliser. As the game entered the final 15 minutes, Williamson hit a shot from 20 yards which flew just over the bar.
It was a surprisingly sparse Vale attack for the last 15 minutes, but City had a big let off with three minutes remaining when substitute Alex Nimely twisted and turned in the area before firing into the side netting.
Another sub Ryan Haynes nearly produced something special for City in the first minute of five added on when he struck a 25 yard free kick that was heading for the top corner, but was saved well by Neal.
The Sky Blues looked to play the ball in the corner, but Vale won it back and Danny N'Guessan had a last gasp shot that Allsop saved and clung on to.
As the Valiants kept pushing they were leaving more holes at the back and City made the game safe with a classic counter.
Madine released O'Brien, who was fouled but kept going. His shot was saved, but Madine was there to stick home the rebound and sent the City fans crazy.
The referee blew up straight after with players, staff and fans all celebrating. It was a win that sends them up to 14th in the SkyBet League One table.
Referee: Mark Heywood
Attendance: 5,321 (708 Away)
CoventryCity: Allsop (GK), Pennington, Martin, Webster, Johnson, Barton, Fleck (Willis 90'), O'Brien, Swanson (Haynes 45), McQuoid (Finch 70'), Madine.
Subs: Burge, Thomas, Jackson, Maddison.
PortVale: Neal, Yates, Dickinson, Veseli, Robertson, McGivern (Birchall 76'), Brown (Dodds 68'), Marshall, Daniel (Nimely 66'), Williamson, N'Guessan.
Subs: Boot, Lloyd, Duffy, Campion