Indexed by:
Abstract:
Complex data-parallel job contains task dependency information defined as Directed Acyclic Graph (DAG). For convenience, the DAG presented data-parallel jobs are named as DAG jobs. The prevalence of DAG jobs in modern data centers has made the scheduling oriented job characterization a big challenge. This paper proposes a deep graph-temporal clustering framework, i.e., DeGTeC, to efficiently categorize DAG jobs leveraging the graphic and temporal information in DAGs. The categorization result can then be naturally used to characterize the resource consumption pattern of DAG jobs. The DeGTeC framework is constructed mainly based on two autoencoders, i.e., TaskAE and JobAE. TaskAE and JobAE contain spectral graph convolutional network (GCN) layers, temporal convolutional network (TCN) layers, and the adaptive pooling layers to help build task embeddings and job embeddings. An extra embedding sorting step takes in the sequential order information and the depth-bias information for job clustering. To our best knowledge, DeGTeC is the first solution to do resource consumption characterization of DAG jobs fully leveraging the task dependencies defined in DAG. Experimental results demonstrate that the DeGTeC framework outperforms the state-of-the-art job resource consumption characterization methods. © 2022 Elsevier B.V.
Keyword:
Reprint Author's Address:
Email:
Source :
Future Generation Computer Systems
ISSN: 0167-739X
Year: 2023
Volume: 141
Page: 81-95
7 . 5 0 0
JCR@2022
ESI Discipline: COMPUTER SCIENCE;
ESI HC Threshold:19
Cited Count:
SCOPUS Cited Count: 7
ESI Highly Cited Papers on the List: 0 Unfold All
WanFang Cited Count:
Chinese Cited Count:
30 Days PV: 0
Affiliated Colleges: