Mobile　phones　are　now　popular　in　current　society　for　its　portability　and　intelligence,　but　excluded　from　life　of　the　hearing-impaired　because　sounds　are　unavailable　for　them.　This　paper　provides　a　framework　of　speech　based　Chinese　Sign　Language　(CSL)　Animation　System　on　Smart　Phones,　which　combines　speech　recognition　and　sign　language　animation　to　make　phones　convenient　for　the　hearing-impaired.　This　paper　recognizes　the　Chinese　speech　into　text　first,　and　then　proposes　a　HMMSVM　model　to　judge　interrogative　expression　from　the　speech　and　rewrites　the　plain　text　as　mark-up　text　according　to　the　rules　of　Chinese　Sign　Language　Mark-up　Language　(CSLML)　which　provides　signs　for　recording　expressive　context　in　the　text,　the　corresponding　animation　is　finally　synthesized　using　key-frame　technology.　The　prototype　system　based　on　the　framework　provided　in　this　paper　gives　plausible　Sign　Language　animation.　©　2018　The　Authors,　published　by　EDP　Sciences.