Brain　disease　diagnosis　based　on　brain　network　classification　has　become　a　hot　topic.　Recently,　classification　methods　based　on　convolutional　neural　networks　(CNNs)　have　attracted　much　attention　due　to　their　ability　to　capture　the　basic　topological　structure　of　the　brain　network.　However,　they　ignore　abnormal　structures　within　modules　caused　by　brain　disease,　which　limits　the　diagnostic　accuracy.　In　this　paper,　we　propose　a　novel　brain　network　classification　framework　based　on　a　CNN　model　capable　of　extracting　modular　features　from　brain　networks　at　the　node　and　whole-network　levels.　More　specifically,　we　first　develop　a　novel　algorithm　to　obtain　the　modular　structure　of　each　node,　which　is　then　fed　into　a　CNN　model　to　extract　the　node-level　modular　features.　Second,　we　minimize　the　harmonic　modularity　of　the　extracted　node-level　features　to　reveal　the　modular　structure　at　the　whole-brain　network　level.　Finally,　we　employ　a　deep　neural　network　to　further　extract　high-level　features　for　the　classification　of　brain　disease.　The　experimental　results　on　a　real-world　autism　spectrum　disorder　dataset　show　that　our　proposed　method　achieves　the　best　accuracy　of　68.55%　and　outperforms　other　common　methods　and　demonstrates　the　discriminant　power　of　the　modular　features　at　multiple　levels.　In　addition,　feature　analysis　based　on　the　trained　framework　reveals　the　associations　between　modular　structures　and　brain　disease,　which　provides　new　insights　into　the　pathological　mechanism　from　the　perspective　of　modular　structures.