Divide timeseries data by first value of each year

I'm working to figure out how to divide each value of each year in a multi-year time series of data by the first value of each year.

A simple example is below:

library(dplyr)
library(lubridate) 

date    <- c('2015-01-01', '2015-01-02', '2015-01-03', '2015-01-04', '2015-01-05', '2015-01-06', '2016-01-01', '2016-01-02', '2016-01-03', '2016-01-04', '2016-01-05', '2016-01-06')
date <- tibble(ymd(date))
numbers <- tibble(1+ seq(0,11)/6) 


data <- bind_cols(date,numbers)
colnames(data)<- c('date', 'number')

firstDay <- data %>%
  group_by(year(date)) %>%
  arrange(date) %>%
  slice(1L) 

colnames(firstDay)[3] <- 'year'

data %>% group_by(year(date)) %>% 
  mutate ( calc = number/firstDay[2,firstDay$year==year(date)] )

I want all data for 2015 to be divided by the value for "number" from Jan 1, 2015. Likewise I want all data for 2016 to be divided by the value for "number for Jan 1, 2016.

I've tried several variations of the last pipe statement but can't find one that works.

Thank you for your help.

jfd118