The 'Castle' used to be Whitstable's manor house, the oldest part being built in the 1790's by Charles Pearson, a London businessman who owned other interests nearby. He bought most of the land around the building, and the title 'Lord of the Manor' from Lord Bolingbroke. The house was only used during the summer months as a residence by the sea.

The next owner was Wynn Ellis, another London merchant whose mistress, Susan Lloyd, lived in the house for a time. Mr. Ellis landscaped the grounds and also built the bell tower. After Wynn Ellis's death in 1875, his son, Sydney, sold the house, the title, and the estate, to the Tankerton Estate company in 1890.

The grand gatehouse to the building was built by the next owner, Thomas Adams, who also renaimed it 'The Towers'.
Albert Mallandain, a paper manufacturer owned the house and the title 'Lord of the Manor' from 1920-1935. He added another block to the building and a panelled billiard room, and during this time, the building and lands were a centre of town social life.

1935 saw the building bought by the Whitstable Urban District Council, to be used as local government offices, at which time, the billiard room was transformed into the council chamber. The grounds were opened as a public park in the same year to mark the Royal Silver Jubilee. In 1972, when Canterbury City Council took over local government, 'The Castle' became a busy community centre, and is now managed by the Voluntary Castle Association.

Many events are held at the castle, including the annual May Day celebrations where all sorts of stalls are erected and live musicians and local artists put on displays for the town celebrations, as traditional local morris dancers entertain the crowds. The castle also hosts many classes and workshops and the public are always welcome to visit and enjoy the beautiful gardens surrounding the main building.