If the question is saying that each item costs a whole number of rupees (is this what "rounded in rupees" means?) then you are right, it is not possible for the shopkeeper to have charged the same prices to each customer (so the fourth option is correct).
This is because in total he has sold two of each item, but he has charged a total of Rs $87$. If he had charged $a$ for each file, $b$ for each notebook and $c$ for each pen then we would have $87=2(a+b+c)$, but $87$ is odd so this is impossible.