Senior Data Engineer - Freelance

Missions

• Own the engineering foundations that make 24/7 data usable, trustworthy, and scalable, including lightweight technical leadership.

• Deliver production-grade ingestion, transformation, data quality, lineage, observability, and secure delivery of data products and calculation-ready datasets.

• Partner with Product Owner, Business Analysts, software engineers, architects, data-platform teams, and country stakeholders.

• Contribute to end-to-end data architecture; document current and target data flows, interfaces, dependencies, and ownership.

• Design and build ingestion pipelines from internal datasets, databases, REST APIs, partner feeds, and controlled file exchanges.

• Integrate heterogeneous country data, isolating source-specific logic via reusable connectors and clear interface contracts.

• Implement batch and, when justified, near-real-time or streaming patterns.

• Define robust handling for schema evolution, late-arriving data, replay, idempotency, backfills, and failed records.

• Collaborate with platform teams to decide when to consume existing data products or shared time-series capabilities instead of duplicating local copies.

• Design scalable structures for high-volume interval data, including customer consumption and asset generation series.

• Define and maintain canonical models for customer, site, metering point, contract, asset, allocation, certificate, time series, matching result, and reporting period.

• Apply appropriate modelling patterns for operational application data, analytical data products, and calculation-ready datasets.

• Address time zones, daylight-saving transitions, interval granularity, units of measure, missing intervals, duplicates, and versioned corrections.

• Optimize storage, partitioning, indexing, and processing performance based on workloads and retention requirements.

• Build tested transformations standardizing source data into calculation-ready consumption and generation curves.

• Enable the Hourly Matching Score and related customer indicators via reproducible, version-controlled datasets and calculation services.

• Partner with business and product specialists to translate calculation rules into explicit technical specifications and automated tests.

• Preserve the distinction between provisional operational measures and validated or certified reporting results.

• Implement automated controls for completeness, validity, timeliness, uniqueness, consistency, and referential integrity.

• Define reconciliation controls between source, ingested, transformed, and reported data.

• Provide end-to-end technical lineage, run metadata, and auditable evidence of source, transformation, and calculation versions.

• Create monitoring, alerting, and operational dashboards covering freshness, pipeline failures, rejected records, and service-level compliance.

• Contribute to incident analysis, root-cause correction, and prevention of recurrence.

• Use Git-based development, peer review, automated testing, and CI/CD to deliver reliable data pipelines.

• Contribute to Infrastructure as Code and environment configuration with DevOps and platform teams.

• Apply security-by-design: least-privilege access, secrets management, encryption, environment segregation, and safe customer data handling.

• Produce maintainable code, runbooks, architecture decision records, data contracts, and operational documentation.

• Participate in Agile ceremonies while maintaining an engineering backlog for reliability, scalability, and technical debt.

Hands-on technical leadership expectations

• Act as the senior engineering reference for data within the 24/7 squad while remaining directly involved in design, coding, testing, and production support.

• Define pragmatic engineering patterns for ingestion, transformation, testing, deployment, monitoring, and recovery, ensuring consistent adoption.

• Lead technical design and code reviews; coach software engineers contributing to data flows without people management duties.

• Challenge tactical duplication and make explicit build-versus-reuse recommendations with B2B data platform and architecture stakeholders.

• Turn ambiguous requirements and fragmented source landscapes into sequenced technical decisions, deliverable increments, and a controlled technical-debt backlog.

• Own key non-functional decisions covering scalability, reliability, security, operability, maintainability, and cost.

Priority data domains

• Customer, legal entity, site, and point-of-delivery reference data.

• Contract parameters, target matching commitments, and reporting calendars.

• Hourly or sub-hourly customer consumption and meter data.

• Renewable or carbon-free asset identity, characteristics, and generation data.

• Allocation of assets and energy volumes to contracts.

• Certificates or Guarantees of Origin and associated evidence.

• Matching results, methodology versions, and customer-reporting outputs.

Required technical capabilities

Essential capabilities

• Advanced Python and SQL for production-grade data processing

• Building and operating data pipelines across APIs, databases, and files

• Ability to translate business concepts into robust technical schemas and data contracts

• Experience handling large-volume and/or time-dependent datasets

• Automated testing, data-quality controls, and reconciliation

• Git, code review, CI/CD, and environment promotion

• Cloud data engineering on Azure or AWS

• Logging, monitoring, alerting, and incident troubleshooting

• Secure data-access patterns and secrets management

Strongly preferred capabilities

• Databricks, Spark, Delta Lake, and Unity Catalog

• Energy, metering, or specialist time-series platforms such as PostgreSQL / TimescaleDB

• Data Lake / Lakehouse architecture

• Real-time or streaming services such as Kafka, Event Hubs, or equivalent

• Terraform or another Infrastructure-as-Code tool

• Data cataloguing, lineage, and governance tooling

• Energy metering, power markets, PPAs, certificates, or Guarantees of Origin

• Power BI or semantic-layer integration

• AI-assisted engineering tools used within approved security practices

Expected deliverables in the first 3-6 months

• Documented map of priority data sources, flows, owners, interfaces, and known gaps for first target countries.

• Minimum technical data model and data contracts for core 24/7 entities, developed with Business Analyst and data architecture support.

• Industrialized ingestion and transformation pipelines for agreed priority consumption and generation sources.

• Automated data-quality, reconciliation, and freshness controls with operational alerts.

• Reproducible calculation-ready datasets supporting Hourly Matching Score and customer reporting.

• Documented recommendation on local time-series storage versus shared platform or TSDB capabilities, including migration implications.

• Runbooks, CI/CD, monitoring, and support procedures sufficient for production operation.

Success measures

• Priority pipelines meet agreed freshness, completeness, and reliability service levels.

• Manual file handling is reduced and, where necessary, controlled through traceable and validated ingestion.

• Every reported result can be traced to its source data and transformation or calculation version.

• New countries and sources onboarded through reusable patterns rather than bespoke end-to-end builds.

• Data incidents detected quickly, explained clearly, and prevented from silently affecting customer reporting.

• Solution reuses shared OneB2B / energy company data capabilities where appropriate and avoids unjustified duplication