Manually setting group colors for ggplot2

I have a data set from which I have to make several dozen plots. The data consists of 30 Groups and several measurements within each group.

Most plots will not use all the groups at once.

My goal is to have one single color pallete for the set of groups such that any given group is the same color across all graphs. In the example below, this would mean that Group C is the same color in Plot 1 and in Plot 2.

My question is how to go about this.

I've tried several variations of scale_fill_manual (and scal_color_manual, when appropriate). While the colors are chosen from the designated pallete, I cannot seem to index the pallette by the group, or some other way to "skip" over the colors corresponding to a missing group

I also thought to try to add the color information as a column of the data being plotted, but I'm unsure how to then use this column as the color for the group.

library(ggplot2)
library(data.table)

# SAMPLE DATA: 
DT1 <- data.table(Name=c("C_sub1", "A_sub1", "A_sub2"), Value=c(2,5,3), Group=c("C", "A", "A"), key="Group")
DT2 <- data.table(Name=c("A_sub1", "B_sub1", "C_sub1", "C_sub2"), Value=c(4,3,6,3), Group=c("A", "B", "C", "C"), key="Group")

# SAMPLE Color Pallette  
ColorsDT <-  data.table(Group=LETTERS[1:5], Color=c("#333BFF", "#CC6600", "#9633FF", "#E2FF33", "#E3DB71"), key="Group")

# Add a column for Color, according to the Group
DT1[ColorsDT, Color := i.Color]
DT2[ColorsDT, Color := i.Color]

# A Basic Plot
simplePlot <- function(DT, tit) 
  ggplot(DT ,aes(x=Name, y=Value, fill=Group)) +
   geom_bar(stat="identity") + xlab("") + ggtitle(tit)
    # Tried sevearl variations of: 
    #  + scale_fill_manual(values=ColorsDT$Color)


# Plot Them
grid.arrange(ncol=2,  simplePlot(DT1, tit="Plot 1"),  simplePlot(DT2, tit="Plot 2"))