The history of the old barn remains largely undocumented. It is believed to have been constructed by Joseph Caldwell, his sons, and their laborers. Joseph acquired the 412-acre farm in 1760 and maintained ownership until his death in 1793. The property remained in the Caldwell family until 1819, when it was sold to George Rust, Jr. In 1831, George D. Smith purchased the property and his family retained ownership until approximately 1910.
During the Civil War, the barn fell victim to Union cavalry operations of late 1864. This destruction was not due to the Smiths’ being Confederate, but rather as part of General Philip Sheridan’s widespread “Burning Raid.” This campaign involved indiscriminate destruction of property and confiscation of crops and livestock throughout the Loudoun Valley, aimed at disrupting John Mosby’s and other Confederate partisans’ activities in the area.