Along　with　the　development　of　machine　learning,　deep　learning,　and　large　language　models　(LLMs)　such　as　GPT-4　(GPT:　Generative　Pre-Trained　Transformer),　artificial　intelligence　(AI)　tools　have　been　playing　an　increasingly　important　role　in　chemical　and　material　research　to　facilitate　the　material　screening　and　design.　Despite　the　exciting　progress　of　GPT-4　based　AI　research　assistance,　open-source　LLMs　have　not　gained　much　attention　from　the　scientific　community.　This　work　primarily　focused　on　metal-organic　frameworks　(MOFs)　as　a　subdomain　of　chemistry　and　evaluated　six　top-rated　open-source　LLMs　with　a　comprehensive　set　of　tasks　including　MOFs　knowledge,　basic　chemistry　knowledge,　in-depth　chemistry　knowledge,　knowledge　extraction,　database　reading,　predicting　material　property,　experiment　design,　computational　scripts　generation,　guiding　experiment,　data　analysis,　and　paper　polishing,　which　covers　the　basic　units　of　MOFs　research.　In　general,　these　LLMs　were　capable　of　most　of　the　tasks.　Especially,　Llama2-7B　and　ChatGLM2-6B　were　found　to　perform　particularly　well　with　moderate　computational　resources.　Additionally,　the　performance　of　different　parameter　versions　of　the　same　model　was　compared,　which　revealed　the　superior　performance　of　higher　parameter　versions.