Rotherham began the season with 17 points deducted following the problems over administration in the summer, but this victory had cleared the deficit and leaves them on plus one.
For most of the first half it seemed they were destined to stay in deficit as they played second fiddle to managerless Grimsby.
The Mariners were much the better side in that first half and put themselves on course for a first win of the season when midfielder Danny Boshell fired them in front on 29 minutes.
It was a superb strike from 20 yards and it gave Andy Warrington no chance as it found the far top corner.
Rotherham struggled throughout the half but their improvement began the moment Reuben Reid equalised four minutes after half-time with his fifth goal of the season.
Alex Rhodes set up the opportunity with an inch-perfect cross to the far post which Reid powered home.
Rotherham simply got stronger and stronger after that boost and on 57 minutes, Mark Hudson fired his first goal of the season, although goalkeeper Phil Barnes would've been disappointed in the way the ball squeezed under him at the near post.
Ian Sharps had a header cleared off the line before Michael Cummins got his first Rotherham goal with a 30-yard volley.
Sub Richie Barker, who recently returned to the club he left four years ago, marked the moment with a typical far post header ten minutes from time from another pinpoint Rhodes centre.