How to set colors for bar plot with multiple layers in R?

I'm using ggplot2 to creat bar plot using three variables "stage", "sex" and "total" from data like below:

    residtraj             sex    stage          perc total
   <chr>                 <fct>  <fct>         <dbl> <int>
 1 born rural live rural female No cancer   0.00725     1
 2 born rural live rural female Early stage 0.02        1
 3 born rural live rural female Late stage  0.0462      3
 4 born rural live rural male   No cancer   0.00625     2
 5 born rural live rural male   Early stage 0.0323      4
 6 born rural live rural male   Late stage  0.0602     13
 7 born rural live urban female No cancer   0.138      19
 8 born rural live urban female Early stage 0.12        6
 9 born rural live urban female Late stage  0.215      14
10 born rural live urban male   No cancer   0.194      62

My code as:

plot <- ggplot(data, aes(sex, total, fill=residtraj)) +
  geom_bar(stat="identity", position="dodge") +
  facet_wrap(~stage) +
  geom_text(aes(label=paste(total,"(",percent(perc),")")),
            position=position_dodge(width=1), hjust=0.5, vjust=-1, size=2.5) +
  ylab("count(%)")+ ggtitle("Distribution of residence by sex and stage")

My question is, how can I set different color according to both "residtraj" and "sex"? (right now the color is automatically assigned only based on the value of "residtraj")