Comcast Careers

Lead Big Data Engineer

Philadelphia, PA
Project and Program Management (Project and Program Management)

Job Description

Business Unit:

Need Experienced Data engineer with 5+ years of experience working with Data Centric Applications and Services with at least 3 years of experience working in Big Data environment

Expert in:
- Build Custom Data Pipelines in Python that Clean, Transform and Aggregate data from many different sources
- Big Data Technologies such as: MapReduce, Hadoop, Spark, HBase, Hive, Elastic Search
- Working with Hadoop and Elastic Search Clusters
- Data Structures and Data Processing Algorithms and Frameworks
- Data Migration, high throughput Data Pipelines
- Massively Parallel Processing using Python Tools
- Big data modelling and ETL Processes
- Ability to analyze Performance Issues in Big Data Environment
- Data Modelling, Data Transfer and Storage, Partitioning, Indexing and caching Techniques
- Analytical views and data visualization
- RDBMS and NoSQL Columnar and Document Storage Systems
- Batch Processing with Petabyte Scale data sets

Well experienced with:
- Large Scale Data Modelling from Big Data perspective
- Big Data Structures in Python
- PyData, Anaconda, numPy, PyTables, DataFrames, Jupyter Notebook
- PyHive, PySpark
- JSON/Parquet Data formats
- Real time streaming with either Spark Streaming of Kafka
- NoSQL, high availability data sources
- Unix Command Line tools
- Unix Shell Scripting
- Develop API's/Tools that help share data with Enterprise
- Debugging in Big Data environment
- Version control Systems: Git/github/BitBucket etc.
- Large Volumes of data Processing and Developing Automated Extraction, Loading and Transformation Processes
- High Throughput Low Latency Systems

Good to have:
- Familiarity with PyPi
- Workflow Management Tools such as Luigi, Apache Airflow, Snowflow or similar
- Experience Developing Models that help Predictions
- Experience with Functional Programming is a Big Plus

Non-Technical Experience:
- Effective Collaborator and Team Player
- Quick Learner and Self-starter with excellent debugging skills
- Ability to foresee scalability issues and make judgements

Comcast is an EOE/Veterans/Disabled/LGBT employer