It is worth looking at the context: Arsène Lupin's accomplice has just mentioned the rumours about Imbert's fortune, which Lupin countered by saying that that fortune is very real ("Que la fortune provienne du vieux Brawford, comme ils le prétendent, ou d’une autre source, j’affirme qu’elle existe.": "Whether it comes from the old Brawford, as some claim, or from another source, it definitely exists").
In addition, Lupin talks of "les prêteurs et les hommes de paille", i.e. the money lenders and the straw men. This suggests that hommes de paille refers to persons that Imbert uses to sign documents on his behalf (see also straw man) on Wikipedia), so outsiders don't know he is behind certain transactions.
The accusations of swindling or fraud published in certain newspapers (and the end of the story) confirm the impression that the Imberts were involved in obscure dealings.