Principal AI Network Engineer - Highrise AI
Hut 8
WorldwideremotePosted 6 days ago
Skill Required
Network-EngineerAI-Infrastructure-EngineerData-Center-Network-EngineerGPU-Infrastructure-EngineerHPC-Network-EngineerSenior-AI-Infrastructure-EngineerPrincipal-Network-EngineerPrincipal-AI-Systems-EngineerAI-HPC-Infrastructure-EngineerAI-ML-Infrastructure-EngineerAI EngineerNetwork EngineerAISystem DesignJMeterQuery OptimizationtroubleshootingEngineeringnetworkingautomationObservabilitydesigningFirmwaresecurityAnsibleTestNGPythonLinuxCloudNode.jsandFulltime
Key highlights
- Experience level: 10+ years of hands-on data center or HPC networking experience
- Notable requirement: Expertise in Ethernet/InfiniBand, RDMA/RoCE, and 200/400/800G networking
- Key benefit: Opportunity to design and pitch initiatives to serial entrepreneurs and seasoned executives
- Work arrangement: Hybrid or remote depending on location (with required travel)
Role overview
Highrise AI is a GPU-native infrastructure platform built for modern AI workloads, serving as the compute and operations layer for its parent company, Hut 8 Corp. The company delivers scalable GPU compute across dedicated clusters, managed cloud, and confidential environments, combining bare-metal performance with full-stack orchestration. Highrise is seeking a principal-level, hands-on AI Network Engineer to lead network fabric bring-up, advanced configuration, performance tuning, and technical acceptance for large-scale GPU clusters. The role involves taking networks from physical installation to commercially ready operation and serving as a Tier 3 escalation point for complex production networking issues.
Responsibilities
- Lead network commissioning of materially complete AI clusters, including switch initialization, configuration validation, host connectivity, and readiness for production workloads.
- Configure and integrate high-speed Ethernet and/or InfiniBand compute fabrics, including RDMA/RoCE, alongside storage, management, and out-of-band networks.
- Validate topology, routing, redundancy, addressing, port mappings, optics, cabling, and interoperability across switches, NICs, and supporting systems.
- Implement and refine network automation, configuration management, firmware/OS baselines, and repeatable fabric deployment procedures.
- Partner with compute and storage engineers to resolve host-to-fabric integration issues and ensure end-to-end cluster readiness.
- Execute fabric qualification and acceptance testing for link health, error rates, bandwidth, latency, congestion, resilience, and scaling behavior.
- Troubleshoot and tune RDMA transport, congestion control, QoS, PFC/ECN, routing, and switch/NIC settings as appropriate to the deployed fabric.
- Analyze NCCL and distributed workload results with compute engineers to isolate communication bottlenecks and improve multi-node performance.
- Use telemetry, switch counters, fabric management platforms, and diagnostic tools to identify degraded links, optical issues, packet loss, or unexpected congestion.
- Own networking acceptance evidence, defect remediation, configuration sign-off, and readiness recommendations for commercial service.
- Provide senior technical escalation for production incidents involving fabric instability, host connectivity, network performance, routing, or persistent hardware faults.
- Lead deep troubleshooting and root cause analysis across switches, NICs, optics, cabling, and network operating systems; coordinate vendor engineering escalations.
- Support network upgrades, expansions, failover testing, controlled configuration changes, and post-maintenance validation.
- Develop diagnostic automation, monitoring standards, runbooks, and operational handoffs for NOC, SRE, and on-site technicians.
- Collaborate with infrastructure engineering, OEMs, networking vendors, site operations, and customer-facing teams to meet deployment and service milestones.
- Provide concise technical reporting on fabric health, acceptance results, outstanding defects, operational risks, and recommended improvements.
Requirements
- Bachelor's degree in computer science, engineering, networking, or related discipline, or equivalent practical experience.
- 10+ years of hands-on data center or high-performance networking experience, including deep troubleshooting and technical leadership.
- Demonstrated success deploying and operating large-scale, low-latency networks for AI, HPC, cloud, or similarly demanding distributed systems.
- Strong expertise in Ethernet and/or InfiniBand fabrics, RDMA/RoCE, routing and switching, and multi-tier data center network architectures.
- Experience with high-speed 200/400/800G networking, NICs, optical interconnects, performance testing, and network telemetry.
- Proficiency with Linux networking and network automation using Python, Ansible, or equivalent tooling.
- Expert-level problem solving across physical, link, transport, and application-facing network performance layers.
- Strong ownership of deployment outcomes, acceptance criteria, operational reliability, and technical documentation.
- Ability to make independent technical decisions while collaborating effectively with senior compute, storage, and systems engineers.
- Clear communication with engineering leadership, customers, site teams, OEMs, and external network vendors.
Nice to have
- Familiarity with NVIDIA networking and fabric-management tools is highly desirable.
Benefits
- Directly influence the performance, reliability, and production readiness of large-scale AI clusters.
- Work alongside experienced infrastructure engineers and help define technical standards for a growing global AI infrastructure platform.
- Work with bright, driven peers from a range of educational and professional backgrounds including software development, energy, engineering, entrepreneurship, investment banking, private equity, and management consulting.
- Design and pitch new products, services, and other initiatives to a leadership team consisting of serial entrepreneurs and seasoned executives and backed by a board of directors consisting of industry veterans of energy, finance, and government.
- Debate ideas and alternatives in a truly meritocratic setting where the learning curve is steep and the lessons come from both senior and junior members of the team.
- Build a lifelong network of friends and professional connections at the cutting-edge intersection of technology, energy, and infrastructure.
Additional details
- This is a full-time, engineering role supporting large-scale AI compute infrastructure across multiple sites.
- Travel to data centers for cluster commissioning, acceptance, and complex troubleshooting will be required.
- Flexibility to assist with critical incidents or maintenance activities outside regular business hours is expected.
- This role is hybrid or remote depending on location.
- Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.
- Originally posted on Himalayas