This plot reminded me of the movie "Being There" starring Peter Sellers. I looked it up and it turns out this film was adapted from a book of the same title published in 1970: _Being There_ by Jerzy Kosinski.
Possibly the book you are after? Though he didn't really have "major troubles adjusting" after he left the mansion. Quite the opposite!