The shavings come in part from the extruder design: there is no fillet on the pulling in side, the filament goes up in a sharp angle and is dragged over a rather sharp edge. It also brushes against the leadscrew.
To help with the shavings, you should alter the filament path to try and have a flatter angle than the 90°. A simple rod that pushes out the filament to come in at 80° might already reduce the number of shavings.
More might be reduced by opening the entry hole of the filament to have a 1-2 mm phase around it, altering the angle of the edge that the filament runs against to a much lower one and reducing the ability to shave off flakes. I did swap my extruder for an aluminium one, and it has this phase, and since then I have had little to no shavings at that spot anymore.