`\K` marks the start of the region to color.
See `man pcrepattern` for details.
> The escape sequence `\K` causes any previously matched characters not to be included in the final matched sequence.
That means you need:
ls -lAF | grep --color -P '^\S+\s+\S+\s+\K\S+\s+\S+'