After a short hike, you will reach the small village of Kibune. I recommend to ignore Kifune Shrine for now and instead walk about 10mins uphill until you reach Hirobun to take a long lunch break.
If you want to have lunch/dinner at Hirobun, you will need to make a reservation online through their website beforehand: https://hirobun.co.jp/ (only available in Japanese, the plugin of Google Translate will help you make a reservation)
Hirobun definitely has the best kawadoko, restaurants that are built over the cooling river streams in summer. A reservation guarantees you 2hrs of access to their stunning kawadoko area. This was one of my absolute favorite summer experiences!
Just on the opposite side of the road you can find the middle shrine of Kifune Shrine, make sure not to miss it!