Data Engineer (Top Engineering Colleges)

A leading blockchain products company specializing in the development of decentralized applications, prediction market platforms and payment infrastructure. Products are used by users in 90+ countries. Founders are IIT, IIM & ISB alumni, who have previously built scalable tech and data-driven products for multiple startups across industries and have a cumulative work experience of 40+ yrs.

About the Role

The Data Engineer will build and own the data backbone of our prediction market trading terminal. You will ingest real-time odds from prediction markets (Polymarket and Kalshi) and sportsbooks (Pinnacle), pull on-chain CLOB data, scrape stats from sources like HLTV and ESPN, and unify everything into a low-latency data layer that powers our trading UI, AI edge models, and automation engine. This is a high-ownership IC role where you architect the entire pipeline—from raw ingestion to the serving layer that traders interact with in real time.

Key Responsibilities

• Own the entire data engineering function end-to-end — from ingestion to serving layer — as a high-autonomy individual contributor.
• Build real-time and batch ingestion pipelines for prediction market APIs (Polymarket Gamma/CLOB, Kalshi), sportsbook odds feeds (Pinnacle), and stats sources (HLTV, ESPN, flashscore).
• Design and implement Medallion architecture (Bronze → Silver → Gold) for market data, on-chain orderbook snapshots, historical odds, match results, and player/team statistics.
• Build the feature store that feeds our AI edge models — Elo ratings, Bradley-Terry map-veto probabilities, Bayesian calibration signals, and Kelly sizing inputs.
• Set up WebSocket listeners and streaming infrastructure for live odds movement, in-play probability updates, and orderbook depth changes with sub-second latency targets.
• Build nightly batch pipelines for model retraining data — historical odds vs. outcomes, walk-forward backtesting datasets, and P&L reconciliation across exchanges.
• Set up cloud infrastructure (AWS/GCP), orchestrate jobs (Airflow/Prefect/cron), and implement monitoring, alerting, and data-quality checks across all pipelines.
• Build data APIs and caching layers that serve the trading terminal frontend with normalized, low-latency market data across all supported exchanges.
• R&D on new data sources, scraping strategies, and tooling to continuously expand market coverage and improve data freshness.

Qualifications

• 2+ years of hands-on Data Engineering experience building production pipelines at scale.
• Strong Python (Pandas, asyncio, aiohttp, requests, BeautifulSoup/Scrapy) and advanced SQL.
• Experience with PostgreSQL, Redis/DragonflyDB, AWS (S3, Lambda, RDS) or GCP equivalents.
• Hands-on experience with WebSockets, streaming data, and real-time event-driven architectures.
• Proficiency in REST API integration, webhook setup, and web scraping at scale (rate limiting, proxy rotation, anti-bot handling).
• Experience with workflow orchestration tools (Airflow, Prefect, or Dagster) and CI/CD on Linux/Docker.
• Solid understanding of database design, Medallion/lakehouse architecture, and data modelling for analytics.
• Engineer from Top IITs, NITs, BITS and other top colleges
• Ability to self-direct execution, diagnose and fix issues proactively without supervision.
• Obsessive data quality mindset, ensuring correctness at a gut level.
• Scrappy problem-solving skills to find workarounds and ship fixes for breaking APIs or changing rate limits.
• Habit of clear documentation for schema definitions, pipeline DAGs, and failure runbooks.
• Comfort with fast iteration and building extensible pipeline architectures.
• Strong cross-functional communication skills to explain technical concepts to non-data stakeholders.

Benefits & Perks

• Receive weekly feedback to help you grow quickly and understand what it takes to move to the next level.
• High-autonomy individual contributor role with full ownership of the data pipeline.