We are a leading trading platform that is ambitiously expanding to the four corners of the globe. Our top-rated products have won prestigious industry awards for their cutting-edge technology and seamless client experience. We deliver only the best, so we are always in search of the best people to join our ever-growing talented team.
We're building out our observability practice and need a senior engineer who can own it end to end. This is a hands-on role. You'll design and operate the telemetry stack that gives our engineering teams real visibility into production — across a hybrid AWS and on-premise environment, at scale.
Responsibilities:
Own the full observability stack: metrics (VictoriaMetrics), logs (OpenSearch), and traces (OpenTelemetry) — from pipeline design to day-2 operations.
Architect and run VictoriaMetrics cluster topology (vmstorage/vminsert/vmselect), including vmagent scraping, remote write configuration, vmalert rules, and cardinality control.
Operate OpenSearch clusters: index lifecycle management (ISM), hot-warm-cold architecture, shard tuning, and ingest pipelines via Data Prepper.
Build and maintain OTEL Collector pipelines — receivers, processors, exporters — and instrument services across Java, Python, and JS/TS stacks (auto and manual).
Run Kafka as the telemetry transport layer (OTEL Collector → Kafka → backends), including topic design, partition strategy, consumer group lag monitoring, and throughput tuning for high-volume telemetry.
Manage log shipping infrastructure using Fluent Bit, Vector, or Fluentd; define structured logging standards and field normalization across services.
Build Grafana dashboards and alerting that engineers actually use — clear, actionable, with well-structured variables and thresholds.
Work with platform and application teams to improve sampling strategies (head/tail), batching, and context propagation across distributed services.
Contribute to incident response, post-mortems, and reliability improvements driven by observability signals.
Mentor engineers on observability practices, tooling, and structured logging standards.
Requirements:
6+ years in a DevOps, SRE, or platform engineering role, with at least 2 years focused on observability tooling at production scale.
Deep hands-on experience with VictoriaMetrics (or Prometheus) — MetricsQL/PromQL, exporters, service discovery, remote write, downsampling, and retention management.
Solid OpenSearch or Elasticsearch skills: cluster operations, Query DSL, ISM policies, and ingest pipeline design.
Production experience with OpenTelemetry: Collector configuration, OTLP, context propagation, and instrumentation across multiple languages.
Strong Kafka skills — producer/consumer patterns, consumer group management, Kafka Connect, Schema Registry, and JMX-based monitoring. Strimzi experience a plus if you've run Kafka on Kubernetes.
Proficiency with log shippers (Fluent Bit, Vector, Fluentd) and structured log parsing/normalization.
Working knowledge of Kubernetes (operators, Helm), Argo CD/GitOps, and Terraform/Ansible.
Comfortable in a hybrid AWS + on-prem environment; solid understanding of networking as it applies to scraping and shipping pipelines.
Scripting ability in Bash or Python for automation and tooling.
Strong communication skills — you can explain observability tradeoffs clearly to engineers and non-engineers alike.
English proficiency.