For 4% of people, color can go much further in making us feel or behave a certain way. Some people are able to associate taste with a number, color with a name, food with words, feel a specific letter, etc. This is called synesthesia, which means having joined sensation such as colored hearing. It's not a disorder but a rare and different way to remember things, it's considered a trait.
Richard E. Cytowic explains what synesthesia is and how people sense numbers, letters, names, and more.