Requirements Gathering : Lead the requirements gathering process for data-related projects, working closely with stakeholders to understand and document their needs. Data Pipeline Development : Design, develop, and maintain data pipelines that enable efficient data extraction, transformation, and loading (ETL) processes using Oracle and MS SQL. Batch Processing : Implement and manage batch processing jobs to support daily banking operations, ensuring timely and accurate data processing. Data Integration : Work with Kafka to manage real-time data streaming and integration with other core banking systems. Project Collaboration : Collaborate with the development team, business analysts, and other stakeholders to ensure data requirements are met and projects are delivered on time. Risk Management : Identify potential risks in data projects and escalate issues or suggest contingency plans as necessary. Progress Reporting : Provide regular updates on data project progress and track key deliverables to ensure alignment with project goals. Task Execution : Complete assigned tasks within established timelines, adhering to the team's development and quality standards.Education : Bachelor’s Degree in Computer Science, Information Technology, Data Science, or a related field, or equivalent work experienceData Architecture Design : Contribute to the design of data architectures that support scalable, reliable, and efficient data processing systems. ETL Development : Develop ETL processes to load and transform data from various sources into structured formats for analysis and reporting. Data Streaming : Utilize Kafka for real-time data streaming, ensuring reliable and low-latency data flow between systems. Performance Tuning : Optimize data pipelines and batch processing jobs to improve system performance and reduce processing times. Data Quality Assurance : Implement data validation and quality checks to ensure the accuracy and consistency of data across systems. Documentation : Maintain comprehensive documentation of data processes, architectures, and system configurations to support ongoing maintenance and development. Continuous Learning : Stay updated with the latest trends and technologies in data engineering, continuously improving technical skills. BAU Support : Provide support for Business as Usual (BAU) operations, troubleshooting and resolving data-related issues as they arise.
Data Engineer • Kuala Lumpur, MY