Nebius AI is an AI-centric public cloud platform specifically crafted to serve AI models for training and inference.
Our mission is to help ML practitioners concentrate on their core jobs, while DevOps, MLOps, and infrastructure-related tasks are handled by us. The idea is to build an ML-specific cloud platform covering the entire ML lifecycle from A to Z: from data preparation and labeling to ML training and inference.
We recognize the potential of ML and AI technologies and aim to provide our future users with the perfect environment to train and fine-tune their models. We are committed to delivering the best user experience and excellent customer support.
Data center in Europe: Our own data center in Finland features server racks designed in-house for ML-specific high loads, with power-efficient solutions, including a free-cooling system.
500+ professionals: Our mature team of Engineers has a proven track record in developing sophisticated cloud and ML solutions and designing cutting-edge hardware.
At Nebius, we're on a mission to harness the power of massive data, and we're looking for an innovative and passionate Engineer experienced in Apache Spark internals to join our team. Our platform, YTsaurus, operates with exabytes of data, and we've recently made this powerful tool open-source. You'll be at the forefront of integrating Apache Spark with YTsaurus, creating an efficient data handling ecosystem.
The unique feature of Spark over YTsaurus, or SPYT as we call it, is its deep low-level integration between Apache Spark compute and YTsaurus storage. This integration allows for efficient processing by utilizing metadata. SPYT supports YTsaurus transactions and uses knowledge of table sorting to eliminate the shuffle phase during JOIN operations. Additionally, SPYT employs YTsaurus as an execution environment for launching Spark clusters in a cloud manner, enabling dozens of SPYT clusters to operate simultaneously within YTsaurus.
For more detailed overview of SPYT, have a look at couple of talks by active and former members of YTsaurus SPYT special interest group:
SPYT is actively used by both internal and commercial users of YTsaurus in Nebius and outside our company. Also, we must define the place of Spark in our next-gen AI platform based on top of YTsaurus. To accomplish these tasks, we need a passionate Spark expert ready for challenges and not afraid of taking responsibility.
You’re welcome to work in our offices in Amsterdam and Belgrade, hybrid or remotely.
It would be an added bonus if you had:
Does this sound like the challenge you've been looking for? If so, we invite you to join us!
Nebius — современная IT-компания, помогающая создавать собственные локальные облачные платформы крупным B2B-бизнесам. Nebius предоставляет не только технологии, но и готовую к запуску бизнес-модель, включая инструменты для поддержки, продаж и маркетинга.
