Posted today · be early
Data Architect – Dynamic Pricing & Decision Intelligence Platform
TechBiz Global
WorldwideremotePosted 1 day ago
Skill Required
Data-ArchitectureData-EngineeringCloud-Data-ArchitectureData-LakehouseReal-Time-ArchitectureData-ArchitectData-Platform-ArchitectData-Analytics-ArchitectPrincipal-Data-ArchitectData-Model-ArchitectData-Infrastructure-ArchitectAI-Data-ArchitectSenior-Data-ArchitectData-Platform-Architecture-ManagerLead-Data-Architectdata engineeringDesign PatternsKafkaEngineeringDatabricksR ProgrammingCryptographySnowflakeMLflowBigQuerysecurityGraphQLAirflowSparkPythondesignFlinkNeo4jAzureCloudMachine LearningjQueryAPIsdbtETLFulltime
Key highlights
- 10+ years of Data Architecture experience.
- Remote work location.
- Design and govern the complete data ecosystem for the ValueX platform.
- Scalable to support 50M+ customer profiles and real‑time event streams.
- Department: Data & AI Engineering.
- Posted on Himalayas.
Role overview
TechBiz Global is recruiting a Data Architect for a client to design and govern the complete data ecosystem of the ValueX platform. The architect will be responsible for data ingestion, processing, modeling, storage, orchestration, and governance across multiple modules, ensuring scalability for 50 million+ customer profiles, real‑time event streams, and ML‑driven decisioning while focusing on performance, cost optimization, and maintainability.
Responsibilities
- Define the end-to-end data architecture across batch, streaming, and real-time layers.
- Design Customer 360 and Offer 360 data models, including feature store and historical layers.
- Establish logical, physical, and semantic data models to enable segmentation, scoring, and orchestration.
- Define data contracts for Kafka topics, API payloads, and Adobe/CDP integrations.
- Set up data versioning and lineage frameworks to track data provenance.
- Architect data ingestion pipelines from multiple telco sources: OCS, CRM, Kenan, Billing, DWH, Adobe AEP, Pricefx, and external APIs.
- Define patterns for real-time event ingestion (recharge, offer purchase, balance check).
- Standardize data access through APIs or data products for downstream modules.
- Design connectors for cloud storage (e.g., S3, Delta Lake) and integration middleware (e.g., n8n, KNIME).
- Define and enforce data quality, lineage, catalog, and access policies.
- Establish metadata management frameworks (e.g., DataHub, Collibra, Amundsen).
- Set up data validation and DQ frameworks (Great Expectations, Deequ).
- Govern data partitioning, schema evolution, retention, and archiving strategies.
- Ensure compliance with data privacy and regulatory standards (e.g., PDPA, GDPR, local telecom data policies).
- Design for high performance and cost‑efficient scalability (25M → 50M → 75M customers).
- Optimize compute/storage balance across environments (Dev, UAT, Prod).
- Define data lakehouse optimization strategies (Z‑Order, Delta caching, compaction).
- Monitor and manage query performance, cluster sizing, and job orchestration costs.
- Work closely with Data Engineers, Data Scientists, and Application Developers to ensure architectural alignment.
- Lead architecture review boards and maintain data design documentation (ERD, flow diagrams, schema registry).
- Serve as technical liaison between business stakeholders, data teams, and platform vendors (Databricks, Adobe, Pricefx).
- Provide best practices and design patterns for model deployment, retraining, and data lifecycle management.
Requirements
- 10+ years in Data Architecture and Data Platform Design.
- Expertise in Data Architecture‑lakehouse design, data modeling (dimensional, normalized, semantic), and schema management.
- Experience with ETL/ELT and orchestration tools such as Databricks, Snowflake, dbt, Airflow, AWS Glue, Azure Data Factory.
- Proficiency in streaming and real‑time technologies including Apache Kafka, Spark Streaming, Kinesis, Flink.
- Ability to design Customer 360, Offer 360, Transaction 360, and Feature Store data models.
- Hands‑on experience with cloud platforms: AWS (S3, Glue, Lambda, EMR), Azure (ADF, Synapse), or GCP (BigQuery, Dataflow).
- Knowledge of storage and compute formats like Delta Lake, Parquet, Iceberg, Snowflake.
- Familiarity with data quality and governance tools such as Great Expectations, Deequ, DataHub, Collibra.
- Strong programming and scripting skills in Python, SQL, PySpark, YAML.
- Competence in API and integration design using REST, GraphQL, Kafka Connect, JSON schema.
- Understanding of security and compliance concepts: IAM, encryption (KMS), access control, masking, PDPA compliance.
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related fields.
Nice to have
- Telecom industry experience (recharge, balance, offer, churn, usage data).
- Experience integrating with Adobe Experience Platform (AEP) or Pricefx.
- Knowledge of KNIME or n8n for workflow orchestration.
- Familiarity with AI/ML pipelines and MLOps frameworks (MLflow, SageMaker).
- Exposure to knowledge graphs (Neo4j, GraphFrames) for segmentation and recommendation.
- Certifications in AWS/Azure Data Architect, Databricks Certified Data Engineer, or Snowflake Architect.
Additional details
- Location: Remote.
- Department: Data & AI Engineering.
- Originally posted on Himalayas.
- Recruitment performed by TechBiz Global on behalf of a top client.
- The role supports ValueX platform modules: Customer Segmentation, Business Decisioning & Offer Assignment Engine, Real‑Time Offer Orchestration Module (ROOM), ML/AI Model Management & Simulation Engine.
- Architect must ensure scalability for 50 million+ customer profiles, real‑time event streams, and ML‑driven decisioning with focus on performance, cost optimization, and maintainability.