Plastic (e.g. polyethylene) is made mostly of hydrocarbon chains. So is oil (both lipids (organic) and inorganic alkanes (paraffin, etc.)). Both of these substances are hydrophobic, so it's hard to get them to mix into water--and you are washing your plastics with water, not DMSO or something, right? But long hydrocarbon chains are happy to stick to each other, as you can tell by looking at melting and boiling points of long alkanes (the more you want to stick, the hotter you have to be before you melt/boil).
On the other hand, it's not as though oil somehow knows that plastic was made from oil. It's simply a property of the chemical interactions. Other surfaces that are lipophilic will have the same hard-to-wash property.
Incidentally, there is research (example) into materials that are both lipophobic _and_ strongly hydrophobic, because almost nothing likes to stick to these and thus the surfaces are "self-cleaning".