post-training & llmops engineer.

Your center of gravity is LLMOps and large-scale training systems: standing up and optimizing multi-node training across cloud and managed ML platforms, and making the difference between a job that "runs" and a job that runs fast, stably, and cost-efficiently at scale. Alongside that, you will work directly with post-training engineers and researchers across the full post-training stack (SFT, RFT/RL, continual pretraining) — setting up the environment, data, and parallelism strategy for fine-tuning runs, improving the systems that couple trainers, inference engines, and rollout workers, and wiring the resulting expert models into company's drug-discovery workflows, interfaces, and tools. 

about the job.

1. Design, provision, and operate large-scale distributed training environments for LLM post-training across multi-GPU, multi-node clusters.
2. Stand up and optimize training on Kubernetes-based and managed ML platforms, including AWS SageMaker HyperPod / EKS, plain cloud Kubernetes (GKE, AKS, self-managed), Tencent TiOne, and similar platforms; abstract away platform differences so researchers can move between them with minimal friction.
3. Configure and tune model and data parallelism strategies — data, tensor, pipeline, sequence/context, and expert (MoE) parallelism — to maximize throughput and fit models to available hardware.
4. Increase training efficiency and minimize training cost through GPU utilization tuning, mixed-precision/quantization-aware training, communication/overlap optimization, optimal batch and sequence packing, spot/preemptible capacity strategies, and right-sizing of clusters.
5. Own high-performance data and storage pipelines for training and benchmarking, including S3 / EFS / FSx for Lustre (and equivalents), data sharding/streaming, caching, and throughput tuning so data loading never bottlenecks the GPUs.
6. Engineer robust checkpointing, fault tolerance, and recovery (including elastic and resumable training) so long multi-node runs survive node failures and preemptions.
7. Build and maintain the environment and tooling around training jobs: container images, dependency/environment management, job schedulers, queueing, autoscaling, and reproducible run configuration.
8. Build observability for training: GPU/throughput/cost dashboards, run telemetry, profiling, and alerting; diagnose stragglers, hangs, OOMs, optimization issues, and performance regressions.
9. Establish MLOps/LLMOps best practices — CI/CD for training and serving, infrastructure-as-code, experiment tracking, artifact/model registries, and reproducibility standards.
10. Design, run, and improve post-training workflows for LLMs, including SFT, RFT/RLVR/RLAIF, and continual pretraining for scientific and reasoning-heavy use cases.
11. Build and optimize the infrastructure that couples trainers, inference engines, asynchronous rollout workers, and evaluation systems for RL-based post-training.
12. Support the data-to-benchmark loop: help convert internal drug-discovery data into datasets usable for model training and prospective benchmarking, and operate the infrastructure for running benchmark/evaluation suites at scale.
13. Analyze failures in fine-tuning runs, diagnose optimization or systems bottlenecks, and turn experimental findings into robust engineering improvements.
14. Help integrate post-trained expert models into production drug-discovery workflows, serving stacks, and internal tools, in collaboration with platform and product teams.

skills and experience required.

1. M.S., Ph.D., or equivalent relevant experience in Computer Science, Machine Learning, Distributed Systems, or a related quantitative/engineering discipline.
2. 5+ years of hands-on experience spanning ML infrastructure / LLMOps and LLM training, or closely related areas.
3. Deep, hands-on expertise with Kubernetes for large-model training, including at least one managed/large-scale platform such as AWS SageMaker HyperPod / EKS, GKE/AKS, Tencent TiOne, or comparable; comfortable across multiple platforms rather than locked to one.
4. Strong practical understanding of distributed-training parallelism — data, tensor, pipeline, sequence/context, and expert parallelism — and how to combine them for throughput and memory efficiency.
5. Hands-on experience with distributed-training frameworks and stacks such as PyTorch (DDP/FSDP), DeepSpeed/ZeRO, and the Hugging Face ecosystem; proven ability to debug and optimize multi-node runs rather than only launching baseline scripts.
6. Fluency with cloud storage and high-throughput data systems for ML — S3, EFS, FSx for Lustre (and equivalents) — including data-loading optimization and storage/throughput tuning.
7. Proven track record of increasing training efficiency and reducing training cost on real workloads (utilization, scaling efficiency, capacity strategy).
8. Demonstrated experience setting up the environment, sequence, and data for post-training of open models, and operating the infrastructure for benchmarking post-trained model performance.
9. Practical understanding of post-training methods — SFT and RL approaches such as GRPO, PPO, DPO, and reward modeling — and the systems implications of RL training loops (trainers, inference engines, rollout workers).
10. Strong skills with containers and orchestration (Docker, Kubernetes) and infrastructure-as-code.
11. Strong Python skills and solid engineering habits around reproducibility, automation, CI/CD, observability, and debugging at scale.