Data Engineer

To support the build and modernization of the Data Hub within the client ecosystem— an enterprise data ecosystem — by developing, automating, and sustaining data pipelines for reporting, performance measurement, and analytics. 🎯 Key Responsibilities -Design, build, and maintain automated, scalable data pipelines for performance and reporting datasets. -Refactor legacy pipelines to meet enterprise data standards and improve performance.  -Implement robust data quality routines (validation, anomaly detection, schema checks).  -Enhance orchestration and error handling for workflow reliability. -Support infrastructure tasks like environment refreshes, upgrades, and disaster recovery testing. -Ensure compliance with metadata, lineage, classification, and documentation standards. -Collaborate on datasets for executive reporting and strategic planning across the Personal and Commercial Operations (PCO), ODIA (Operational Data, Information and AI). [Apply now at https : / / my.hiredly.com / jobs / jobs-malaysia-talentspothub-management-consulting-job-data-engineer]

Core Technical Skills Required -3+ years in hands-on data engineering -Tools and Languages : -PySpark, Python, SQL  -Hadoop ecosystem : Spark, Hive, HDFS -Shell scripting, Linux environments   -Query tuning with Spark SQL, Trino, Hive -Deployment experience across dev / test / prod -Working with partitioned datasets, Parquet, columnar storage -Strong grasp of data governance, metadata, and validation   ⭐ Preferred / Bonus Skills -Experience with event-driven or real-time processing (e.g., Kafka) -Familiarity with vector databases, LLM-supportive architectures -Tools : Apache Airflow, OpenShift (OCP4), AWS (for orchestration / scheduling) We welcome candidates with the skillset above to immediately drop an email to nisha@talentspothub.com for more information and confidential conversation.