Weld　penetration　identification　is　a　long-standing　and　challenging　problem　due　to　the　spatial　limitation　in　sensing　the　back-side　of　weld　joints　in　practical　welding,　and　the　key　characteristic　information　of　welding　is　difficult　to　be　defined　and　extracted　in　the　complex　welding　process.　This　paper　proposes　an　end-to-end　deep　learning　approach　to　predict　the　weld　penetration　status　from　top-side　images　during　welding.　In　this　method,　a　passive　vision　sensing　system　with　two　cameras　is　developed　to　monitor　the　top-side　and　back-bead　information　simultaneously.　Then,　the　weld　joints　are　classified　as　three　classes　i.e.　under,　desirable　and　excessive　penetration　depending　on　the　back-bead　width.　Taking　the　top-side　images　as　inputs　and　corresponding　penetration　status　as　labels,　an　end-to-end　convolutional　neural　network　(CNN)　is　designed　and　trained　where　the　features　are　defined　and　extracted　automatically.　Testing　experiments　demonstrate　92.70　%　as　the　classification　accuracy.　In　order　to　increase　the　accuracy　and　training　speed,　a　transfer　learning　approach　based　on　residual　neural　network　(ResNet)　is　developed.　This　ResNet-based　model　is　pre-trained　on　ImageNet　dataset　to　process　a　better　feature　extracting　ability　and　its　fully-connected　layers　are　modified　based　on　our　own　dataset.　The　experiments　show　that　this　transfer　learning　approach　can　decrease　the　training　time　with　the　prediction　accuracy　improving　to　96.35　%.