Data Science and Analytics · Pune, India

Big Data Engineer- Scala

Capco · Pune, India · Posted Nov 3, 2026

This posting comes from Capco. Confirm the details on their site before applying.

Overview

We are looking for an experienced Big Data Engineer with strong expertise in Scala and Apache Spark to design, develop, and optimize scalable data-processing solutions. The ideal candidate should have hands-on experience with distributed data platforms, large-scale ETL/data pipelines, and big data technologies.

Responsibilities

  • Design, develop, and maintain scalable Big Data applications and data pipelines.
  • Develop high-performance data-processing solutions using Scala and Apache Spark.
  • Build and optimize ETL/ELT pipelines for large volumes of structured and unstructured data.
  • Work with technologies such as Hadoop, HDFS, Hive, Kafka, and Spark SQL.
  • Perform Spark job performance tuning and optimization.
  • Develop reusable and maintainable data-processing frameworks.
  • Troubleshoot production issues related to data pipelines and distributed applications.
  • Implement data-quality checks, monitoring, and error-handling mechanisms.
  • Collaborate with data architects, analysts, business teams, and other engineering teams.
  • Follow software engineering best practices, including Git, code reviews, testing, and CI/CD.

Requirements

  • Strong hands-on programming experience in Scala.
  • Strong experience with Apache Spark / Spark SQL.
  • Good understanding of Big Data and distributed computing concepts.
  • Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive.
  • Experience building large-scale ETL/data engineering pipelines.
  • Strong SQL skills.
  • Experience working with Kafka or similar messaging/streaming technologies.
  • Understanding of data formats such as Parquet, Avro, and JSON.
  • Experience with performance optimization and troubleshooting of Spark applications.
  • Good understanding of software development lifecycle and version-control systems.
  • Experience with AWS, Azure, or GCP.
  • Knowledge of Databricks.
  • Experience with Airflow or similar orchestration tools.
  • Exposure to Delta Lake or other modern lakehouse technologies.
  • Knowledge of Docker, Kubernetes, and CI/CD pipelines.

Great fits start here.

Whether you're looking for your next role or your next team member, give people and possibilities a place to meet.