The　emergence　of　large　language　models　(LLMs)　has　provided　robust　support　for　application　tasks　across　various　domains,　such　as　name　entity　recognition　(NER)　in　the　general　domain.　However,　due　to　the　particularity　of　the　medical　domain,　the　research　on　understanding　and　improving　the　effectiveness　of　LLMs　on　biomedical　named　entity　recognition　(BNER)　tasks　remains　relatively　limited,　especially　in　the　context　of　Chinese　text.　In　this　study,　we　extensively　evaluate　several　typical　LLMs,　including　ChatGLM2-6B,　GLM-130B,　GPT-3.5,　and　GPT-4,　on　the　Chinese　BNER　task　by　leveraging　a　real-world　Chinese　electronic　medical　record　(EMR)　dataset　and　a　public　dataset.　The　experimental　results　demonstrate　the　promising　yet　limited　performance　of　LLMs　with　zero-shot　and　few-shot　prompt　designs　for　Chinese　BNER　tasks.　More　importantly,　instruction　fine-tuning　significantly　enhances　the　performance　of　LLMs.　The　fine-tuned　offline　ChatGLM2-6B　surpassed　the　performance　of　the　task-specific　model　BiLSTM+CRF　(BC)　on　the　real-world　dataset.　The　best　fine-tuned　model,　GPT-3.5,　outperforms　all　other　LLMs　on　the　publicly　available　CCKS2017　dataset,　even　surpassing　half　of　the　baselines;　however,　it　still　remains　challenging　for　it　to　surpass　the　state-of-the-art　task-specific　models,　i.e.,　Dictionary-guided　Attention　Network　(DGAN).　To　our　knowledge,　this　study　is　the　first　attempt　to　evaluate　the　performance　of　LLMs　on　Chinese　BNER　tasks,　which　emphasizes　the　prospective　and　transformative　implications　of　utilizing　LLMs　on　Chinese　BNER　tasks.　Furthermore,　we　summarize　our　findings　into　a　set　of　actionable　guidelines　for　future　researchers　on　how　to　effectively　leverage　LLMs　to　become　experts　in　specific　tasks.