With　the　advance　of　computer　technology　and　smart　device,　many　technologies　and　applications　have　been　developed　to　enhance　the　efficiency　of　human-computer　interaction　(HCI).　For　human,　the　hand　is　a　natural　and　direct　way　in　communication.　Hand-held　Object　Recognition　(HHOR),　which　is　to　predict　the　label　for　the　object　people　hold　in　hand,　can　help　machines　in　understanding　the　environment　and　people＇s　intentions.　However,　it　has　not　been　well　studied　in　the　community.　So,　in　this　paper,　we　proposed　a　novel　feature　fusion　based　method　for　hand-held　object　recognition　with　RGB-D　data.　First,　the　skeleton　information　is　used　to　initially　locate　the　object　and　with　depth　map　we　extract　object　region　in　a　region-growing　manner.　Then　on　the　corresponding　object　point　cloud,　we　use　Multiple　Kernel　Learning　(MKL)　to　fuse　the　shape　feature　with　color　feature　to　obtain　the　advantages　of　them.　Specially,　we　collected　a　dataset,　which　contains　12800　video　frames　of　16　categories　and　each　frame　captures　the　visual　image,　depth　map　and　user　skeleton　data.　The　experiment　shows　promising　results　in　both　segmentation　and　recognition.　Copyright　2014　ACM.