Building of the stone castle was started by the de Roos family in 1186 to replace a wooden structure that was built on the site around 1120. In 1478 Edward de Roos sold the castle to Richard, Duke of Gloucester (later Richard III) but restored to Edward de Roos by King Henry VII after Richard III was killed on Bosworth Field. It was besieged in 1644 during the English Civil War, and after its capture by the Parliamentarians it was deliberately damaged to prevent any further use as a stronghold.