Does UDP can be exposed to stateful analysis? From protocol theory we know that TCP is stateful protocol. Different stateful packet filters do filtration based on connection state. I.e. it can distinguish responses and replies. It's obviously for TCP, and also ICMP (as an answer on UDP e.g., Frag Needed and so on), but what is about pure UDP exchange? How can one distinguish UDP response and UDP reply without deep L7 analysis?

It is really guessing based on the source and destination IP addresses and ports, along with the timing. After a period of no traffic between those addresses and ports, it is considered done. Different observers will have their own algorithms to determine when a UDP exchange is closed based on the timing. There is no standard for that because UDP is explicitly connectionless. Things like firewalls are moving to deep packet inspection, and how that is done is proprietary to the vendor.

Many UDP exchanges are small and short, e.g. a DNS request and reply. Some, e.g. VoIP, will be almost constant in each direction, while some, e.g. video, will be constant primarily in one direction. Almost all UDP exchanges will either be small and short, or they are real-time traffic that is a constant stream. That is changing as things like QUIC come into play.