Location:
CN-Shenzhen-HyQ
Shift:
Standard - 40 Hours (China)
Scheduled Weekly Hours:
40
Worker Type:
Permanent
Job Summary:
Lead the architecture, design, and technical strategy for the company's open-source data platform. Make the end-to-end technical vision — from ingestion to analytics — while mentoring a team of data platform engineers. This is a hands-on leadership role responsible for design the blueprints and write critical-path code.
Job Duties:
Responsibilities
Architecture & Strategy
- Define the overall data platform architecture on OpenShift, covering the full lifecycle: ingestion (Kafka/Redpanda), compute (Spark, Flink, Trino), storage (MinIO, Iceberg), catalog (Polaris/Gravitino), orchestration (Airflow/Dagster), and serving (StarRocks/ClickHouse)
- Design the Data Lakehouse architecture: open table formats (Iceberg, Delta Lake), catalog federation, multi-engine interoperability
- Architect the real-time data path: Kafka topic design, schema registry (Apicurio/Confluent), stream processing (Flink/Spark Structured Streaming)
- Design the analytical serving layer: OLAP engine selection (StarRocks/ClickHouse/Doris), materialized view strategy, query federation with Trino
- Design multi-tenancy, data security (Ranger/OpenPolicyAgent), RBAC, and governance (DataHub/Atlas) across all platform layers
- Set technical standards for Helm chart design, CI/CD pipelines, and GitOps (ArgoCD/Flux) workflows
- Own data lifecycle management: hot/warm/cold tiering across MinIO, compaction, retention, and GDPR/deletion compliance
Technical Leadership
- Lead a team of 3–6 data platform engineers; run code reviews, design reviews, and sprint planning
- Establish engineering best practices: testing, observability (OpenTelemetry, Prometheus, Grafana, Loki), incident response, runbooks
- Partner with data engineering, analytics, and business teams to translate their needs into platform capabilities
- Define platform SLOs/SLIs across freshness, latency, availability, and durability; drive the on-call rotation and incident post-mortems
Hands-on Engineering
- Build and maintain the core OpenShift infrastructure: operators, Helm charts, namespaces, RBAC, network policies
- Develop Spark and Flink job frameworks, tuning guides, workload scheduling (YuniKorn/Volcano)
- Implement the data mesh or data product model: domain ownership, self-serve data infrastructure, federated governance
- Performance-optimize across the stack: Kafka throughput, Spark shuffle, Iceberg compaction, Trino query planning, OLAP cold reads
Required Skills & Experience
- 10 + years in data/platform engineering, with 3+ years as architect or tech lead
- Deep Kubernetes/OpenShift: operators, Helm, CRDs, admission webhooks, SCC, multi-tenancy at production scale
- Expert-level streaming: Kafka/Redpanda internals (partitioning, consumer groups, exactly-once semantics), schema registry, Kafka Connect
- Expert-level batch & streaming compute: Spark internals (Catalyst, Tungsten, AQE, shuffle, dynamic allocation), Flink or Spark Structured Streaming
- Strong open table format knowledge: Iceberg and/or Delta Lake — table format internals, metadata evolution, partitioning transforms, maintenance, catalog integration (Polaris, Gravitino, Nessie, or Unity Catalog)
- Hands-on with federated query engines: Trino (coordinator/worker architecture, connector ecosystem, fault-tolerant execution mode)
- Solid OLAP knowledge: StarRocks, ClickHouse, or Apache Doris — primary key models, materialized views, query optimization
- Production MinIO/S3-compatible storage at scale: erasure coding, multi-site replication, IAM, tiering
- Pipeline orchestration: Airflow (DAG design, sensors, dynamic task mapping) or Dagster
- Infrastructure-as-code (Terraform/Pulumi/Crossplane) and GitOps (ArgoCD/Flux)
- Fluent in at least one JVM language (Scala/Java) and Python
Nice to Have
- Experience deploying dbt (data build tool) with Trino or Spark adapters for data transformation
- Data governance hands-on: DataHub or Apache Atlas for lineage, glossary, and metadata search
- Data security: Apache Ranger or OpenPolicyAgent for fine-grained access control
- Open-source contributions to any of the above projects
Company Introduction:
ITD SZ
港交所科技（深圳）有限公司，是2016年12月28日于深圳市前海自贸区成立的外商独资企业。
作为港交所的技术子公司，港交所科技（深圳）有限公司主要是为集团及其附属公司提供计算机软件、计算机硬件、信息系统、云存储、云计算、物联网和计算机网络的开发、技术服务、技术咨询、技术转让；经济信息咨询、企业管理咨询、商务信息咨询、商业信息咨询、信息系统设计、集成、运行维护；数据库管理、大数据分析；以承接服务外包方式提供系统应用管理和维护、信息技术支持管理、数据处理等信息技术和业务流程外包服务。
