Talent.com
This job offer is not available in your country.
Spark Engineer

Spark Engineer

Qi GroupPetaling Jaya, Selangor, Malaysia
1 day ago
Job description

We are seeking a highly skilled

Spark Engineer

to design, build, and optimize large-scale data processing systems using

Apache Spark

  • . The ideal candidate will have deep expertise in distributed data processing, ETL pipelines, and performance tuning for high-volume data environments. You will collaborate with data scientists, analysts, and engineers to ensure scalable, reliable, and efficient data solutions.

Key Responsibilities :

  • Design, develop, and maintain big data solutions using Apache Spark (Batch and Streaming).
  • Build data pipelines for processing structured, semi-structured, and unstructured data from multiple sources.
  • Optimize Spark jobs for performance and scalability across large datasets.
  • Integrate Spark with various data storage systems (HDFS, S3, Hive, Cassandra, etc.).
  • Collaborate with data scientists and analysts to deliver robust data solutions for analytics and machine learning.
  • Implement data quality checks, monitoring, and alerting for Spark-based workflows.
  • Ensure security and compliance of data processing systems.
  • Troubleshoot and resolve data pipeline and Spark job issues in production environments.
  • Required Skills & Qualifications

  • Bachelor's degree
  • in Computer Science, Engineering, or related field (Master's preferred).

  • 3+ years
  • of hands-on experience with

    Apache Spark

    (Core, SQL, Streaming).

  • Strong
  • programming skills

    in

    Scala

    Java

    , or

    Python

    (PySpark).

  • Solid understanding of
  • distributed computing concepts

    and

    big data ecosystems

    (Hadoop, YARN, HDFS).

  • Experience with
  • data serialization formats

    (Parquet, ORC, Avro).

  • Familiarity with
  • data lake and cloud environments

    (AWS EMR, Databricks, GCP DataProc, or Azure Synapse).

  • Knowledge of
  • SQL

    and experience with

    data warehouses

    (Snowflake, Redshift, BigQuery is a plus).

  • Strong background in
  • performance tuning

    and

    Spark job optimization

  • Experience with
  • CI / CD pipelines

    and

    version control

    (Git).

  • Familiarity with
  • containerization

    (Docker, Kubernetes) is an advantage.

    This is a 12-month contract appointment, with the potential for extension or conversion to a permanent position subject to performance and business needs.

    Create a job alert for this search

    Engineer • Petaling Jaya, Selangor, Malaysia

    Related jobs
    • Promoted
    Senior Applications & Proposal Engineer

    Senior Applications & Proposal Engineer

    SulzerSelangorMalaysia, Selangor, Malaysia
    Sulzer offers you a wealth of opportunities to drive your career into the direction that you want, depending on your skills and interests. We invite you to apply if you are ready to take the next st...Show moreLast updated: 30+ days ago
    • Promoted
    Equipment Engineer – Semiconductor Finishing

    Equipment Engineer – Semiconductor Finishing

    Renesas ElectronicsSelangorMalaysia, Selangor, Malaysia
    The Equipment Engineer (Finishing Process) is responsible for maintaining, troubleshooting, and improving equipment performance for vision machines, packaging machines, and AMR systems to ensure st...Show moreLast updated: 4 days ago
    • Promoted
    Planning Engineer

    Planning Engineer

    Siemens EnergyPetaling Jaya, Selangor, Malaysia
    A Snapshot of Your Day : You will be responsible for preparing and checking schedules, documentation, and equipment lists, coordinating with Siemens Energy head office and local sub-suppliers, and p...Show moreLast updated: 4 days ago
    • Promoted
    • New!
    Senior Applications and Proposal Engineer, PS

    Senior Applications and Proposal Engineer, PS

    SulzerSubang Jaya, Selangor, Malaysia
    Senior Applications and Proposal Engineer, PS.BD, product, applications) and market / customer demands.Involves preparing detailed cost estimates, engaging in contract negotiations, and collaborating...Show moreLast updated: 3 hours ago
    • Promoted
    Advanced Control Applications Engineer

    Advanced Control Applications Engineer

    ExxonMobilKuala Lumpur, Kuala Lumpur, Malaysia
    Select how often (in days) to receive an alert : .At ExxonMobil, our vision is to lead in energy innovations that advance modern living and a net-zero future. As one of the world’s largest publicly tr...Show moreLast updated: 30+ days ago
    • Promoted
    Be proactive & apply! Structural Engineer opportunities in Malaysia (Local contract)

    Be proactive & apply! Structural Engineer opportunities in Malaysia (Local contract)

    SBM OffshoreKuala Lumpur, Kuala Lumpur, Malaysia
    Press Tab to Move to Skip to Content Link.Select how often (in days) to receive an alert : .Job Family : Structural Topsides. As an Internal Employee we value the contribution you make to SBM Offshore....Show moreLast updated: 30+ days ago
    • Promoted
    SAP Engineer

    SAP Engineer

    LenovoPetaling Jaya, Selangor, Malaysia
    Lenovo is a US$69 billion revenue global technology powerhouse, ranked #196 in the Fortune Global 500, and serving millions of customers every day in 180 markets. Focused on a bold vision to deliver...Show moreLast updated: 30+ days ago
    • Promoted
    SAP EWM Developer

    SAP EWM Developer

    Westernacher ConsultingKuala Lumpur, Malaysia
    We’re not just hiring—we’re building the future of SAP Supply Chain Execution.Westernacher Consulting is expanding our growing team in South East Asia supporting Global SAP Projects, and we’re look...Show moreLast updated: 30+ days ago
    • Promoted
    Senior Process Engineer

    Senior Process Engineer

    Petron Malaysia Refining & Marketing BhdPort Dickson, Negeri Sembilan, Malaysia
    At Petron, we are not just in the business of oil, we are also in the business of fueling lives.Petron Malaysia is an emerging and rapidly evolving Asian oil company. It is part of Petron Corporatio...Show moreLast updated: 30+ days ago
    • Promoted
    • New!
    Welding Mechanical Engineer | Subang Jaya

    Welding Mechanical Engineer | Subang Jaya

    Euro Potential Sdn BhdSubang Jaya, Selangor, Malaysia
    This range is provided by Euro Potential Sdn Bhd.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more. Develop and improve welding procedures, consuma...Show moreLast updated: 3 hours ago
    • Promoted
    Facilities Planning Engineer (Cyberjaya)

    Facilities Planning Engineer (Cyberjaya)

    Talent RecruitCyberjaya, Selangor, Malaysia
    About the job Facilities Planning Engineer (Cyberjaya).Take primary responsibility, working very closely with APAC design team plan for constructability of New Data Center Build / Major fit-out, fr...Show moreLast updated: 30+ days ago
    • Promoted
    Senior SAP EWM Consultant

    Senior SAP EWM Consultant

    Westernacher ConsultingKuala Lumpur, Kuala Lumpur, Malaysia
    Westernacher Consulting is expanding our Delivery Centre in South East Asia to support a growing portfolio of SAP projects in the Region and Beyond. With over 55 years of excellence, 30 years as an ...Show moreLast updated: 30+ days ago
    • Promoted
    • New!
    SAP Developer - S4 / HANA Implementation

    SAP Developer - S4 / HANA Implementation

    Lupus ConsultingSelayang Municipal Council, Selayang Municipal Council, Malaysia
    We are seeking an experienced SAP Developer to join a developer team in an upcoming S / 4HANA implementation.The project will adopt the Keep the Core Clean approach, requiring adherence to SAP-recomm...Show moreLast updated: 3 hours ago
    • Promoted
    • New!
    Senior Subsea Engineer

    Senior Subsea Engineer

    McDermott International, LtdKuala Lumpur, Kuala Lumpur, Malaysia
    The Senior Subsea Engineer uses best practices and knowledge of internal or external issues to improve the Subsea discipline within McDermott. They will act as a resource for colleagues with less ex...Show moreLast updated: 3 hours ago
    • Promoted
    Sr Engineer, Continuous Improvement

    Sr Engineer, Continuous Improvement

    AnsellSeremban, Negeri Sembilan, Malaysia
    At Ansell, we stay two steps ahead of workplace risk to deliver innovative safety solutions that enhance people's quality of life. As a global leader in protection solutions, we design and develop a...Show moreLast updated: 4 days ago
    • Promoted
    • New!
    Equipment Process Engineer

    Equipment Process Engineer

    Sensata TechnologiesSubang Jaya, Selangor, Malaysia
    We do not sell your information.More information can be found atEquipment Process Engineer page is loaded## Equipment Process Engineerlocations : Subang Jaya, Malaysiatime type : Full timeposted ...Show moreLast updated: 3 hours ago
    • Promoted
    • New!
    Engineer - PTM

    Engineer - PTM

    Oman ShellCyberjaya, Selangor, Malaysia
    Engineer - PTM page is loaded## Engineer - PTMlocations : Cyberjaya-Wisma Shelltime type : Full timeposted on : Posted Todayjob requisition id : R Selangor, Malaysia • •Job Family Group : • •Product...Show moreLast updated: 3 hours ago
    • Promoted
    Staff Engineer

    Staff Engineer

    AirAsiaKuala Lumpur, Malaysia
    Job Title : Technology Lead (Agent Portal / Lemon).About AirAsia MOVE AirAsia MOVE is a dynamic Asean focused budget travel company that’s redefining the way people experience travel.As part of the C...Show moreLast updated: 30+ days ago
    • Promoted
    Maintenance Engineer

    Maintenance Engineer

    VeoliaKuala Lumpur, Kuala Lumpur, Malaysia
    Veolia Groupaims to become the benchmark company for ecological transformation.Present on five continents with nearly 220,000 employees, the Group designs and deploys useful, practical solutions fo...Show moreLast updated: 30+ days ago
    • Promoted
    Equipment Engineer – Semiconductor Finishing

    Equipment Engineer – Semiconductor Finishing

    Renesas Electronics CorporationSelangorMalaysia, Selangor, Malaysia
    The Equipment Engineer (Finishing Process) is responsible for maintaining, troubleshooting, and improving equipment performance for vision machines, packaging machines, and AMR systems to ensure st...Show moreLast updated: 2 days ago