Principal Software Quality Engineer

Accountabilities:

Quality Strategy and Test Architecture: Define and own the overall quality strategy for the distributed database platform, including test architecture, coverage models, release criteria, and risk-based testing priorities. Drive quality across server components, client SDKs, database operators, backup tools, and management interfaces.
Distributed Database Testing: Design and execute comprehensive testing strategies covering replication, rebalancing, partitioning, sharding, consistency models, transactions, isolation, secondary indexes, query planning, compaction, backup and restore, snapshots, and cross-datacenter replication.
Fault-Injection and Reliability Testing: Validate cluster behavior under production-like failure conditions, including node crashes, network partitions, clock skew, disk failures, latency, rolling upgrades, mixed-version deployments, and long-running stress tests.
Advanced Debugging and Root-Cause Analysis: Investigate complex system-level defects involving memory access, serialization, encoding and decoding, compression, checksums, and consistency violations. Reproduce issues at source-code level, identify potential root causes, and provide engineering teams with minimal reproducible test cases and actionable technical reports.
Test Automation and Framework Development: Build, maintain, and improve scalable automated testing frameworks using Python, pytest, custom test harnesses, API testing, and end-to-end automation. Ensure reliable execution across shared testing infrastructure and CI/CD environments such as GitHub Actions and Jenkins.
Regression and Infrastructure Reliability: Strengthen regression coverage, reduce flaky tests, investigate inconsistent test results, and develop testing strategies across unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and soak testing.
AI-Agent Integration: Lead the integration of AI agents into existing testing frameworks, enabling automated test planning, execution, log inspection, metrics analysis, defect reporting, and failure triage. Integrate agents with pytest, custom testing harnesses, and MCP-style tool servers.
AI Evaluation and Quality Assurance: Establish evaluation frameworks using golden datasets, graders, and regression suites to measure the reliability, accuracy, and consistency of AI-generated outputs while maintaining appropriate human oversight.
Cross-Functional Collaboration: Participate in product requirement discussions, feature reviews, technical design reviews, and client specification assessments to establish testability requirements early in the development lifecycle.
Technical Leadership and Mentorship: Lead complex debugging and defect-resolution initiatives across quality, client, and server engineering teams. Mentor senior QEs and SDETs, promote effective debugging practices, and establish reusable engineering standards without requiring direct people-management responsibilities.
Continuous Quality Improvement: Convert production incidents and pre-release defects into permanent regression tests, monitoring checks, and preventive measures to improve overall platform reliability.
Requirements:
Professional Experience: 10–18 years of experience in Software Quality Engineering, SDET, or Systems Engineering, with demonstrated Staff or Principal-level technical impact and a strong focus on building quality systems.
Distributed Database Expertise: Extensive hands-on experience testing distributed databases or storage systems, preferably NoSQL platforms such as Cassandra, ScyllaDB, MongoDB, Couchbase, Redis, DynamoDB, or HBase. Experience with distributed relational databases such as CockroachDB, TiDB, YugabyteDB, or Spanner-style systems is also relevant.
Database Testing Knowledge: Strong experience validating replication, partitioning, sharding, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication.
Distributed Systems Fundamentals: Deep understanding of replication protocols, consensus mechanisms, quorum reads and writes, strong and eventual consistency, split-brain scenarios, clock synchronization issues, and failure-recovery semantics.
Systems-Level Debugging: Ability to read, analyze, and debug server-side code written in C, C++, Go, Rust, or Java, tracing complex defects to specific functions and identifying underlying technical causes.
Protocol and Serialization Knowledge: Understanding of communication protocols, serialization formats, and edge cases involving wire formats, MessagePack, Protocol Buffers, Thrift, and compression codecs.
Programming and Automation: Strong Python programming and software automation skills, with proficiency in at least one systems programming language.
Fault-Injection Experience: Practical experience with fault injection, GDB, chaos-style testing, process termination, network partitions, latency and packet-loss injection, disk failures, corrupted inputs, version incompatibilities, and boundary-value testing.
Comprehensive Testing Expertise: Experience across multiple testing layers, including unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and long-running soak testing.
CI/CD and Test Infrastructure: Strong experience with CI/CD platforms such as GitHub Actions, Jenkins, or equivalent tools, including managing and improving shared testing infrastructure.
AI-Agent Testing: Practical experience integrating AI agents into software testing workflows, including agent orchestration, tool integration, prompt design, evaluation methodologies, and integration with real-world testing frameworks.
Technical Communication: Excellent written and verbal communication skills, with the ability to produce clear technical documentation, defect reports, investigation summaries, RFCs, and cross-functional engineering discussions.
Ownership and Leadership: Proven ability to independently lead complex investigations, collaborate across engineering teams, communicate technical findings, and drive defects through to resolution.
Strongly Preferred Skills:
Experience testing client SDKs across multiple programming languages, including Python, Java, Go, C, Node.js, and .NET.
Knowledge of compatibility matrices, upgrade and downgrade scenarios, and behavioral consistency across different client SDKs.
Experience testing backup and restore mechanisms, checkpointing, snapshots, and WAL/journal correctness, including performance testing at terabyte scale.
Expertise in query-engine and secondary-index testing, including query planning, index selection, pagination, boundary conditions, and encoding or collation correctness.
Experience with Kubernetes and database operator testing using tools such as KUTTL, Chainsaw, or envtest.
Cloud infrastructure experience across AWS, GCP, or Azure, including TLS/mTLS, IAM, secrets management, and air-gapped installations.
Security-focused testing experience, including fuzzing, CVE reproduction, and identifying memory-safety issues such as out-of-bounds reads and use-after-free defects in native code.
Benefits:
Full-time, remote working opportunity based in India.
Opportunity to work on a sophisticated distributed database platform and solve challenging systems-engineering problems.
Senior technical individual-contributor role with significant ownership of quality strategy and architecture.
Opportunity to shape and modernize large-scale automated testing frameworks and infrastructure.
Hands-on exposure to advanced distributed systems, database technologies, and fault-injection methodologies.
Opportunity to lead the adoption of AI agents and intelligent automation within software quality engineering.
Collaboration with experienced quality, client, and server engineering teams.
Opportunity to mentor senior engineers and influence engineering practices and technical standards across teams.
Professional growth through exposure to complex database architectures, cloud infrastructure, and emerging AI technologies.