The meaning seems to hold to traditional understandings or color as being about skin tones and various emotions, but there is a slight shift, when he shakes the leaves off... if we understand the changing season hinted at here perhaps he is rejecting other colors and feelings or ideas and reducing it to the " brown brown brown" or in the writers opinion, the more elegant monotonal world. Colors, is illusory, not an implication that the writer prefers variety.