Two　dimensional　Convolutional　Neural　Networks　(ConvNets)　have　been　widely　adopted　as　powerful　models　and　have　achieved　state-of-the-art　performance　in　many　image　related　tasks.　However,　their　extensions　are　still　struggling　for　leading　performance　for　high　dimensional　(HD)　signal　processing,　partially　due　to　the　explosion　of　training　parameters,　greatly　enhanced　computational　complexity　and　memory　cost.　In　this　paper,　we　present　a　simple,　lightweight,　yet　efficient　ConvNet,　called　S-Net,　for　the　HD　signal　processing　by　allowing　a　separable　structure　on　the　ConvNet　throughout　the　learning　process.　It　takes　advantage　of　a　series　of　one　dimensional　convolution　kernels　to　handle　N-dimensional　(ND)　signals.　Thus,　the　presented　S-Net　significantly　reduces　the　training　complexity,　parameters,　and　memory　cost.　The　proposed　S-Net　is　evaluated　on　both　2D　and　3D　benchmarks　CIFAR-10　and　KTH.　Experimental　results　show　that　the　S-Net　achieves　competitive　performance　with　greatly　reduced　computational　and　memory　costs　in　comparison　with　the　state-of-the-art　ConvNet　models.