Engenheiro(a) de Dados Pleno/Sênior AWS – Especialista em Glue, Spark (PySpark) e Data Mesh at IBM Consulting

Apply

Data Engineer with specialization in AWS ecosystem responsible for designing, developing, and maintaining ETL pipelines, working with AWS Glue, EMR, Spark, Python, and related services to support data warehouse and lake architectures, data migration, governance, and analytics dashboards.

Requirements

Education

  • Bachelor's Degree

Experience

  • Proven and practical experience with AWS Glue, Spark (conceptual), PySpark and Python.
  • Knowledge of Data Mesh and distributed data architecture.
  • Experience with AWS Quicksight, S3, RedShift, Aurora, DynamoDB, Glue Catalog, Glue Databrew, Lambda Functions.
  • Experience with batch and streaming data pipelines (Kinesis, Kafka).
  • Mastery of SQL and dataframe manipulation.
  • Experience with tools such as Apache Airflow and dbt.
  • Ability to work in an agile, collaborative and multicultural environment.

Skills

  • AWS Glue
  • Spark (conceptual)
  • PySpark
  • Python
  • Data Mesh
  • AWS Quicksight
  • S3
  • RedShift
  • Aurora
  • DynamoDB
  • Glue Catalog
  • Glue Databrew
  • Lambda Functions
  • Kinesis
  • Kafka
  • SQL
  • Apache Airflow
  • dbt
  • Agile collaboration

Responsibilities

  • Design, develop and maintain scalable ETL pipelines using AWS Glue, EMR, Spark (PySpark) and Python.
  • Build and operate batch and real-time data pipelines, integrating multiple sources and destinations (S3, RedShift, Aurora, DynamoDB).
  • Implement and optimize workflows and dataframes for high performance and reliability.
  • Migrate data using AWS DMS and manage data catalogs with Glue Catalog.
  • Develop dashboards and visualizations with AWS Quicksight.
  • Apply Data Mesh principles for distributed data architecture and governance.
  • Use open source tools such as Apache Airflow, dbt, Spark/Scala.
  • Ensuring data quality, security and governance throughout the lifecycle.
  • Collaborate with multidisciplinary teams to deliver solutions that support advanced analytics and business intelligence.

Technologies

AWS GlueAWS EMRSpark (PySpark)PythonAWS KinesisAWS DMSRedShiftAuroraDynamoDBAWS QuicksightAWS LambdaGlue CatalogGlue DatabrewApache AirflowdbtData MeshS3KafkaSQLSpark/Scala

Share job

See if your resume is ready for this job

See how our AI can optimize your resume and improve your chances for this role.