James and Deborah Brooke's home, Charley Forrest, was in many ways the cradle of Sandy Spring. When they built on the raw frontier in 1728, it was the westernmost framed house south of Canada. Here James Brooke, assisted by brother-in-law John Thomas who arrived a scant year later, built Sandy Spring's first grist mill. Here James and Deborah raised their six children, James, Jr., Roger, Richard, Basil, Elizabeth, and Thomas; from here Brooke acquired other land until he owned some 20,000 acres; here he kept his prized pack of Brooke hounds, brought from England by his grandfather; and here he deeded land for "the people called Quakers" so they could build their Meeting House and bury their dead. Observed historian Roger Brooke Farquhar: "That original house gave good service to the Brookes and others for 185 years, until it was removed in 1913..." Charley Forrest stood on present Brooke Road, on a prominence that gave a sweeping view of Brooke's Sandy Spring.

Granddaughter of James and Deborah, daughter of Roger Brooke IV, Mary Brooke married Thomas Moore in 1791 and settled at Longwood. A bumper crop of Brooke females attracted a significant influx of out side males, bringing diversity to Sandy Spring's early preponderance of Brookes and Thomases.