We suggest that the nonarbitrariness both of synaesthesia and of metaphor (and their directionality) arise because of constraints imposed by evolution and by neural hardware (Ramachandran & Hubbard, 2001a). For example, you say loud shirt but you rarely say red sound; you say sharp taste but rarely bitter touch.
S. Ramachandran and Edward M. Hubbard, The Phenomenology of Synaesthesia, Journal of Consciousness Studies, Vol. 10, No. 8, 2003