You can set up a "session directory" so that some data is stored and, when you exit rtorrent cleanly, you can open it without going through the hashing.
According to the manpage, this can be done using the `-s path` option, so `-s ~/torrentdir` would use that as session directory. But you probably want to set this through `~/.rtorrent.rc` so that you don't have to specify it all the time.
(Sorry for the lack of a working example, I don't have a computer with `rtorrent` set up near me right now.)