Handling it with different user accounts may well be the only possible way since processes do not own any files and can therefore not have a disk quota.
To make it even clearer, at the very best you could manage a quota for the files currently used, should you develop such a kernel patch, but it would still lose its sense to track the files that were written previously and got closed as they are not under its responsibility at all.
Doing such a flawed patch would also result in considerable performance degradation and wouldn't make sense in situations where more than one program opens the same file.
For those and many other reasons, it theoretically simply can not be done properly.