Google 'cyclocross'.
!CX bike in action
The primary difference between a road bike and a CX bike is the size of the tires. You can ride your road bike anywhere your skills will allow. There are some gotcha's though.
Skinny tires only have so much traction. Gravel flats won't be an issue for all but the lightest of race tires, but pinch flats from hitting larger rocks at high speed are a problem.
Dirt and high end road components don't mix well.
But seeing exactly how far you can take a road bike has a long history and many local clubs have "grasshopper" rides that link up old gravel roads. Your bike is capable of much more than you might think. Look up the races Paris-Roubaix and Strada Bianca.