James and Susan and their ten children moved into the new house in 1857 and lived there until 1862. James, a Southern sympathizer, fled Kentucky for Canada in 1862 to escape imprisonment during the Civil War. He was joined by his wife and family in Montreal, where he died of consumption in 1864.

In 1866 the house and 324 acres were auctioned to the Kentucky Agricultural and Mechanical College, the forerunner of the University of Kentucky. The school was at Ashland from 1866 until 1882. During most of that time, the main house served as the residence for the school’s regent, John B. Bowman.

In 1882 the Kentucky A & M was dissolved and the property was sold once again. This time, however, it was sold back into the Clay family. Anne Clay McDowell, Henry Clay’s granddaughter, and her husband, Major Henry Clay McDowell, who was not a relation, but only a namesake of Henry Clay, bought the property and made it their home. It was at this time Major McDowell turned the entire acreage over to horse breeding and the Ashland Stud Farm became known as one of the finest horse farms in the Bluegrass region.

While Major McDowell was changing the farm, Anne was changing the interior of the main house. She oversaw the redesigning of the grand staircase from its original elliptical design to the more fashionable Eastlake style stairway. She also added the back stairway, butler’s pantry, and first indoor plumbing, and introduced gas lighting into the house.

The house and outbuildings underwent an extensive renovation between 1990 and 1991. It was during this time many "discoveries" were made both in and about the house and grounds. Much of what is now on display was brought out of the attic and basement and the interpretation of the house was made more authentic and true to the way the family lived at Ashland.

The house and grounds as we see them today reflect five generations of the Clay family. Ashland's collection represents many aspects of the family’s importance in local and national history from the early 19th century to 1948. Nearly every item in the collection belonged to the Clay family and is original to the estate. This gives the visitor a unique view into the life and times of a great statesman, Henry Clay, and his descendants.