Optimizing Big Data Processing for Efficient Storage, Retrieval, and Analysis Using Deep Reinforcement Learning.

Authors

  • Dr. R. Saravanakumar Saveetha School of Engineering Author

DOI:

https://doi.org/10.54228/mjaret0624012

Keywords:

Big Data Processing; Deep Reinforcement Learning; Data Storage Optimization; Data Retrieval; Data Analysis; Energy Efficiency; Cloud Computing; Edge Computing; Resource Allocation; Adaptive Systems

Abstract

This paper proposes a new method for optimizing big data processing based on deep reinforcement learning (DRL) techniques. We proposed a DRL-based framework that improves the efficiency of storage, retrieval, and analysis for a large amount of data. The framework dynamically adjusts resource allocation and optimizes data flow through clustering based on real-time performance metrics and workload identification. In the experiments, the system is deployed in a simulated data center scale (dc1k) with 1000 nodes, 10 workloads, and 10 petabytes (10 PB) of data. Compared with traditional optimization, the proposed model can save 30% energy consumption, cut the average time for data retrieval by 25%, and boost the running speed of analysis by 20%. The results indicate the adaptive capabilities of the proposed model for different workloads and data types, which could be further evaluated within various big data scenarios.

Downloads

Download data is not yet available.

Author Biography

  • Dr. R. Saravanakumar, Saveetha School of Engineering

    Associate Professor/ Department of Electronics and Communication Engineering

Downloads

Published

2024-08-30

Issue

Vol. 4 No. 3 (2024)

Section

Research Articles(s)

License

License Terms for MJARET

Creative Commons Attributio 4.0 International (CC BY) License:

This license allows for the following:

  1. Sharing — Copy and redistribute the material in any medium or format.

  2. Adaptation — Remix, transform, and build upon the material.

The license is subject to the following terms:

  • Attribution:

    • You must give appropriate credit, provide a link to the license, and indicate if changes were made.
    • Attribution should include the citation of the article, the author's name, and a link to the original work published in MJARET.
    • This must be done in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

  • NonCommercial:

    • The material cannot be used for commercial purposes.
    • Any use of the work intended to provide a commercial advantage or monetary compensation is considered outside the scope of this license.

  • No Additional Restrictions:

    • You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

General Provisions:

  • Understandability: This license can be revoked if you do not comply with its terms and conditions.
  • Public Domain: Where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
  • Other Rights:
    • The license does not cover rights such as publicity, privacy, or moral rights that may affect your ability to use the material as contemplated by the license.
    • Such rights might need to be considered and respected separately.

Disclaimer:

  • MJARET does not provide any warranties with the work. The work is provided "as is" without any representations or warranties, express or implied. MJARET will not be liable for any damages resulting from the use of the work.

How to Cite:

  • Proper attribution for use of the licensed work should follow the standard citation format provided by MJARET, which should include the author(s), the title of the work, MJARET, and the DOI link.