As　a　neuromorphic　vision　sensor　with　ultra-high　temporal　resolution,　spike　camera　shows　great　potential　in　high-speed　imaging.　To　capture　color　information　of　dynamic　scenes,　color　spike　camera　(CSC)　has　been　invented　with　a　Bayer-pattern　color　filter　array　(CFA)　on　the　sensor.　Some　spike　camera　reconstruction　methods　try　to　train　end-to-end　models　by　massive　synthetic　data　pairs.　However,　there　are　gaps　between　synthetic　and　real-world　captured　data.　The　distribution　of　training　data　impacts　model　generalizability.　In　this　paper,　we　propose　a　zero-shot　learning-based　method　for　CSC　reconstruction　to　restore　color　images　from　a　Bayer-pattern　spike　stream　without　pre-training.　As　the　Bayer-pattern　spike　stream　consists　of　binary　signal　arrays　with　missing　pixels,　we　propose　to　leverage　temporally　neighboring　spike　signals　of　frame,　pixel　and　interval　levels　to　restore　color　channels.　In　particular,　we　employ　a　zero-shot　learning-based　scheme　to　iteratively　refine　the　output　via　temporally　neighboring　spike　stream　clips.　To　generate　high-quality　pseudo-labels,　we　propose　to　exploit　temporally　neighboring　pixels　along　the　motion　direction　to　estimate　the　missing　pixels.　Besides,　a　temporally　neighboring　spike　interval-based　representation　is　developed　to　extract　temporal　and　color　features　from　the　binary　Bayer-pattern　spike　stream.　Experimental　results　on　real-world　captured　data　demonstrate　that　our　method　can　restore　color　images　with　better　visual　quality　than　compared　methods.