If you don't manually kill it off, the agent will stay resident in your system.
Aside from being untidy on general principle, this has two potential problems:
1. As Kusanalanda points out it means you will wind up spawning another `ssh-agent` processs every time you log in, which gets kind of sloppy on the process table even if their footprint is probably negligible.
2. More seriously, each of those processes will have your _unlocked_ private keys in their memory. If an attacker were to gain access to your system and exploit a (hypothetical, I'm not currently aware any exist) bug in either kernel memory management or the `ssh-agent` code itself to potentially extract your private keys.
Sure, the last one is low probability, but it's just as easy to kill the process and remove the risk entirely.
Better safe than sorry.