The original Duffus castle was of typical motte-and-bailey design, and was built built before 1150. The wooden keep was replaced by a stone tower some time in the 14th century. At some stage, possibly as early as the 15th century, a large part of the north corner of the tower broke off and slipped part way down the side of the motte, where it still lies today.