The phrase dates back to the early part of the 20th Century. It referred to a possibly apocryphal story. Each month, Charlie Higson's Father would have to go to collect his salary from his work. Grumbling, he would complain that because he had to go out to get his salary it would almost certainly rain.
People began to use the phrase colloquially when they pessimistically thought it was going to rain.