As the manufacturing industry moves toward full intelligence, data has become one of the core production factors for enterprises.
The above-mentioned leading smart home manufacturing companies have continued to build a data-driven system covering the entire chain of R&D, production, marketing, and services in recent years, and the big data platform is the core hub in this system that is responsible for massive data storage, calculation, analysis, and dispatch.
As the enterprise’s business scale rapidly expands and business complexity continues to increase, the 2015 version of the CDH cluster that the enterprise has used for many years has gradually exposed shortcomings such as insufficient technical performance, lack of security capabilities, and limited expansion space. It has become a core bottleneck restricting the release of data productivity. Platform upgrades are imminent.
This big data platform upgrade project faces three core tests: First, business continuity requirements are extremely high. The platform carries a large number of core tasks such as production scheduling, real-time reporting, and supply chain analysis. During the upgrade process, it must ensure that business is uninterrupted, data is not lost, and operations are not interrupted; second, the data volume is huge, and migration
The complexity is high. PB-level historical data is scattered in different business modules. Multiple business teams rely on the platform to run at the same time. They need to complete full data migration within a limited time window and ensure that the data processing results of the old and new platforms are completely consistent. Third, the version span is large, and compatibility and security reconstruction are difficult. From the old CDH
Upgrading from 5.x to the latest CDP (Cloudera Data Platform) not only involves a comprehensive replacement of the underlying engine from MapReduce to Tez/Spark, but also requires the simultaneous introduction of security capabilities such as Kerberos authentication, Ranger fine-grained permission control, and data desensitization, which places extremely high requirements on job transformation and application adaptation.
In response to the above difficulties, the project service team provided a set of systematic rolling migration solutions: Create a new CDP cluster in the new data center, gradually migrate data and workloads from the old cluster to the new cluster according to business scenarios, and verify the data consistency and operation stability for multiple rounds before taking the old equipment offline. At the same time, the idle old server resources are reused to expand the new cluster to minimize project investment costs.
In order to ensure that the migration process is foolproof, the project team conducted multiple rounds of stress testing and simulation verification on the migration path, job compatibility, and data consistency before the official switch. They simultaneously provided job adaptation and transformation plans to each business team, established a strict data processing consistency verification mechanism, and finally formulated a complete traffic switching plan to achieve a smooth rollout without any perception of the business.
After the completion of this upgrade, the enterprise’s big data platform has achieved a multi-dimensional leap in capabilities: First, the performance has been greatly improved. After all offline tasks have been migrated from MapReduce to Tez/Spark, the overall data throughput of the platform has increased by more than 30%, and the processing time of baseline tasks in core areas has been shortened by up to 48%. This has significantly improved the timeliness of data output and can better support real-time needs such as production scheduling and user analysis. Second, storage costs have been significantly reduced, using Hadoop.
The RS-3-2-1024k compression algorithm in EC (Error Correcting Code) technology reduces storage space usage by 40% without reducing data reliability, effectively controlling the cost pressure caused by the continued expansion of data scale; thirdly, security capabilities have achieved a qualitative leap, and Kerberos security has been fully adopted
Authentication realizes mutual trust authentication of various components of big data. Unauthorized machines cannot access clusters and components. With Ranger fine-grained permission management, it supports dynamic column filtering, data desensitization, and customized access policies. All Hadoop components are unified into the security management and control system, fully meeting enterprise-level data compliance requirements.
Industry insiders pointed out that this big data platform upgrade project serves as a benchmark case for the iteration of core data infrastructure in the manufacturing field. Its migration experience of “non-stop business service and zero data error” has a high reference value for the majority of manufacturing companies that are promoting digital transformation. It also provides a replicable computing power base construction path for the domestic manufacturing industry to upgrade from “manufacturing” to “intelligent manufacturing”.
