Draw a line between points of different shapes in ggplot2

Here is some dummy data

d = data.frame(
  x = rep(1:8,2),
  y = c(1,1.2,1.5,2,2.6,2.9,3.1,3.2,0.7,1,1.2,1.9,2.4,2.62,2.95,2.95),
  color = rep(LETTERS[1:2], each=8),
  shape = c(rep("a",6), rep("b",2), rep("a", 5), rep("b",3))
)

I would like to draw a line linking the points in the different colors. Because I would like to keep a space between the line and the point, I am using geom_pointline from the package lemon

My issue is that the line does not connect points of different shapes together. How can I solve this problem?