ETL Jobs

I need a robust, production-ready batch processing pipeline that can reliably pull data from several relational databases, perform the required transformations, and load the cleansed results into our analytics environment on a scheduled basis. Scope • Design and build the end-to-end pipeline architecture—ingestion, transformation, validation, and load. • Optimise for scalability and fault tolerance; orchestration should handle retries and alerting gracefully. • Source systems will be only relational databases for now (PostgreSQL and MySQL are the main ones), but the design should allow new sources to be plugged in later without major refactoring. • All code must be version-controlled and deployable through our existing CI/CD setup (GitHub Actions). Tech ...

I need to build an AI power Business Intelligence platform with dashboard-style SaaS with analytics, NLP querying, conversational analytics, and forecasting. Need a seamless platform which breakdown silos for my clients in the automotive vertical, mainly car dealerships who weave in and out of 10-15 platforms. Need a platform the easily connects with APIs and connects to media advertising and CRM and DMS like Tekion, CDK Global, and Reynolds. Three main goals for the SaaS which has a revenue structure of paid monthly subscriptions. 1) Fast daily data updates, easy to use and customizable dashboard interface for users to see KPIs, MoM, YoY for leads, sales, close rates, and toggle between dealers and choose which dimensions to include for metrics like close rate (Option to choose which lead...

My Terraform code already spins up the MySQL source, VPC, subnets and security groups; Glue can reach the database without additional networking work. What I still need is the Glue side itself—fully defined in Terraform—so the data moves from that MySQL backend into the raw tier of my data lake. Here is what has to happen: • The Glue job (PySpark preferred) connects to MySQL via JDBC, performs the required data cleansing, then writes the output as Parquet files into a specified S3 prefix. • All Glue resources—Job, Connection, IAM role, catalog tables, optional crawler—must be described in Terraform so a single terraform apply recreates the stack in any account. • Partitioning or dynamic frame optimisations that make downstream analytics easier ...

I have an Excel workbook with several sheets of live inventory information that must be migrated into our SAP environment. The file tracks daily progress reporting for each SKU across multiple warehouses, so the solution needs to carry that data across without loss or distortion. What I need: • A repeatable conversion routine—LSMW, BAPI script, or another SAP-approved method—that reads every sheet, maps each column correctly, and uploads the records into the appropriate SAP tables. • Validation checks to confirm row counts and key figures (especially the daily progress metrics) match pre- and post-load. • A concise how-to document so my team can rerun the process whenever new inventory files arrive. Acceptance criteria will be a successful test import...

I run a prediction system, an ERP system, a standalone CRM, several database servers, and a CONTROL SYSTEM / MES on our production floor. Right now each platform lives in its own silo and I constantly need to move files or run manual exports. The immediate goal is true data synchronization—when a customer record updates in the CRM it should appear in the ERP, when production status changes in the MES it should be logged back to the central database, and so on. If this also streamlines processes and improves analytics, that is a welcome bonus, but real-time, bidirectional data consistency comes first. Here is what I need from you: • Assess the current architecture and identify the safest connection points (APIs, message queues, ETL jobs, middleware, etc.). • Build and co...

Looking for an Experienced Data Engineer Project Description I am looking for a skilled Data Engineer to help design, build, and maintain reliable data pipelines and infrastructure. The work may include collecting data from different sources, cleaning and transforming it, and storing it in a database or data warehouse. You may also work with APIs, cloud services, ETL workflows, and large datasets. The ideal candidate should have strong experience with Python, SQL, data modeling, and ETL/ELT pipelines. Experience with AWS, GCP, or Azure is preferred. Knowledge of tools such as Airflow, Spark, Kafka, Snowflake, BigQuery, or Databricks would be a plus. Please include the following in your proposal: A short introduction about your Data Engineering experience Examples of similar projects ...