The　number　of　people　suffering　from　ophthalmic　diseases　is　increasing　with　the　population　aging.　Many　studies　have　been　proposed　to　automatically　identify　diseases　to　reduce　the　risks　of　further　retinal　damage.　However,　most　of　existing　methods　mainly　used　a　single　type　of　dataset　to　solve　the　specific　medical　task,　which　is　not　clinically　practical　in　the　real-world　scenarios.　In　this　paper,　we　propose　a　multi-task　deep　learning　network　based　on　multi-types　datasets　to　automatically　recognise　different　ophthalmic　diseases.　Specifically,　we　first　collect　a　multi-label　dataset　from　the　retinal　fundus　images　and　related　diagnostic　reports.　Then,　we　propose　a　feature-fusion　network　to　extract　image　and　semantic　retinal　information　from　multi-types　datasets.　Finally,　a　multi-stream　models　is　designed　to　integrate　different　specific　features　and　realize　the　multiple　disease　detection.　In　this　way,　multi-types　datasets　based　features　are　fully　extracted　in　a　multi-task　learning　manner.　Experiments　on　our　real-world　dataset　show　that　our　proposed　network　significantly　improve　the　classification　performance　of　the　model　for　ophthalmic　diseases.