A　discriminative　deep　belief　network　(DDBN)　based　on　the　Fisher　criterion　is　used　here　to　calculate　the　super-vector　feature　space　of　speech　signals.　The　network　extracts　the　feature　codebook　of　the　speaker　that　is　superior　to　the　one　from　the　traditional　deep　belief　network　(DBN)　algorithm　for　multi-speaker　clustering　and　segmentation.　Evaluations　on　the　multi-speaker　audio　stream　corpus　generated　from　the　TIMIT　database　show　that　the　speaker　segmentation　algorithm　based　on　the　DDBN　with　the　Fisher　criterion　performs　better　than　the　traditional　Bayesian　information　criterion　(BIC)　method　and　the　DBN　method.