The full expression is “cold enough to freeze the balls off a brass monkey”.
The expression supposedly dates back to a time when cannon balls were stored on the decks of ships in pyramid-shaped stacks held in place by a brass frame around the base. This frame was called a ‘monkey’, and when it got very cold, the brass monkey would contract, causing the stacks of cannon balls to collapse.