The return of Kenny Arthur in goal should have provided a boost for Town, but he suffered a hamstring injury in the pre-match warm-up and had to be replaced by Richard O'Donnell.
Fleet took the lead through a header from Gareth Seddon just before the break, and doubled their advantage after 59 minutes through Nathan Pond.
The goals were deserved for the visitors, who had looked more dangerous despite the Mariners controlling possession.
Alan Connell pulled one back shortly afterwards with a neat shot from the edge of the area. Town pushed for an equaliser but struggled to create opportunities. The best chance was shot wide by Connell.