Refuge of a President, Madison House was built initially by Brookeville founders Richard and Deborah Thomas soon after their marriage in 1783. It passed to Caleb and Henrietta Bentley in 1798. Bentley became Brookeville's first postmaster in 1802 and also ran a store. Legend holds that Henrietta gave up her bedroom to the President and slept on the floor.

Samuel Riggs built Pleasant Hill, the earliest Riggs home, in Brookeville. He fought in the Revolution beside Richard Brooke in Zadok Magruder's Montgomery County Militia.