Indexed by:
Abstract:
In this article, we design and implement an air quality data processing system based on big data technology, which leverages Flume to acquire data, Kafka to cache data, and Storm to process data, and finally save the results to the in-memory database called Redis and distributed database called HBase. In Redis, only the latest data of each monitoring point is saved, and the original data is deleted periodically to reduce the memory load. An application is implemented by which people can get real-time data from Redis and query historical data from HBase. Through experiments, it is proved that querying real-time data from Redis is faster than relational database. The system solves the issues of poor storage capacity and slow query speed when using relational database in the big data environment, and improves the efficiency of data processing significantly. © 2018 IEEE.
Keyword:
Reprint Author's Address:
Email:
Source :
Year: 2018
Page: 1846-1850
Language: English
Cited Count:
WoS CC Cited Count: 0
SCOPUS Cited Count:
ESI Highly Cited Papers on the List: 0 Unfold All
WanFang Cited Count:
Chinese Cited Count:
30 Days PV: 9
Affiliated Colleges: