Plot several segments with a partially common legend

Say that I have the following data frame:

line_df <- data.frame(x_ini = c(0.55,2.55,1.55,5.55,7.55),
x_end = c(1.45,3.45,2.45,6.45,8.45),
y = c(-0.5,-0.5,-1.5,-2.5,-2.5),
category = c("red","red","blue","green","green"))

I want to plot an horizontal segment for each row of the dataframe which extremes would be (x_ini[i],y[i]) and (x_end[i],y[i]) and the legend is given by line_df$category.

I have tried with geom_segment, segments, abline and lines but I can't manage to set the legend properly and plotting the segments with the same instructions. I have it after a plot done with ggplot of a different data frame (so not after a plot instruction) and it seems this is also causing me problems.

Any suggestion on how could I do this? Happy to modify the data frame to other more suitable structure but I want to avoid calling the same instruction several times for each one of the segments.

Thanks!

EDIT

The code right before it looks like

previous_df <- data.frame(letter=c("x","y","y","z","x","p"),number=c(9,3,2,1,5,7),labeldf=c(1,15,12,15,1,12))

ggplot(previous_df,aes(labeldf,number,fill=letter)) +
    geom_bar(stat="identity",position='dodge')

So the problem is that I can't add another ggplot to it and if I add directly the geom_segment (mentioned in the answer by Nova) I get errors regarding the fill option.