These rules are not set in stone. Should men take hats off in a fast food restaurant picking up take out? Probably not. At an outdoor cafe? If you are sitting at a table, its best to err on the side of being overly polite and remove your headwear.
For men, there is no distinction in hat styles when considering etiquette.
Women, on the other hand, have a little more freedom to wear as they damn well please. If they are donning fashionable hats, it is typically acceptable to wear them indoors or outdoors as long as your are not in the work environment. They should also never block anybody's view, at church or the theater, for instance. When women wear unisex hats (for example, baseball caps), they should basically follow the same guidelines for etiquette for men.