The castle was built on the site of a medieval fortified manor, the current structure was designed by architect Thomas Hopper for the wealthy Pennant family.