The name 'Warld's End' reflects the position of the house at what was the harbour's edge. The land now in front was reclaimed from the sea in the 19th century. The house is one of the oldest in the town.

The owner of the original building on the site was John Gordon of Glenbuchat who was a leading Jacobite commander in the ill-fated 1745 rebellion. Bonnie Prince Charlie's struggle to re-establish the Stuart throne came to an end at Culloden. Many Jacobites fled to Norway and this house was the last thing they saw of their homeland - it was their Warld's End.

The house was seized from John Gordon by the Crown and it lay unoccupied until it was sold by auction, in what we call a public roup, in 1766. It was bought by the next generation of the Glenbuchat family, Charles Gordon, and rebuilt the following year in the Georgian style.