Running both tires at the same PSI could be your problem. In a comment you stated you are running both at 70 PSI.
Tires of different width will typically have different design pressures. Larger tires run at lower pressure. Run both tires at the max pressure for _that_ tire. On the sidewall will be an operating pressure range for _that_ tire. Run both tires at the maximum for _that_ tire.
If you are running both tires at the same pressure you are penalizing the smaller tire that is designed to run at a higher pressure.
The size of the contact patch is PSI times weight. So if you have 140 lbs on the tire at 70 psi the contact patch is 2 square inches. The more narrow tires must have a longer contact path to achieve that same 2 square inches. The more narrow tire must deform more. That deformation is rolling resistance. More deformation is more rolling resistance.
This link discusses resistance, deformation, and pressure Rolling Resistance