Principal Software Quality Engineer

This role is for one of Weekday’s clients

Min Experience: 10+ years Location: Remote (India), Bengaluru, Karnataka, India JobType: full-time

We are looking for an experienced Principal Software Quality Engineer to define and drive the quality strategy for a distributed database platform. This is a senior individual-contributor role requiring deep expertise in distributed database testing, systems engineering, debugging, and test automation. The ideal candidate will have hands-on experience working with distributed database or storage systems, preferably NoSQL platforms. They should be comfortable reading server-side code, tracing complex defects to their root cause, designing fault-injection scenarios, and collaborating with QE, client, and server engineering teams to ensure product reliability. The role will also lead the integration of AI agents into testing frameworks, enabling automated test generation, failure triage, and evaluation of agent reliability. Requirements Key Responsibilities Quality Strategy and Test Architecture - Own the overall quality strategy for the distributed database platform, including test architecture, coverage models, release gates, and risk-based prioritization. - Drive quality across server components, client SDKs, operator and backup tooling, and management interfaces. - Design comprehensive testing strategies for replication, rebalancing, partitioning, sharding, consistency models, transactions, isolation, secondary indexes, query planning, compaction, backup and restore, snapshots, and cross-datacenter replication. - Establish reliable coverage for critical database operations and ensure quality requirements are incorporated early in the development lifecycle. Distributed Systems and Fault-Injection Testing - Validate cluster correctness under challenging production-like conditions, including node failures, network partitions, clock skew, disk-full scenarios, slow disks, rolling upgrades, mixed-version deployments, and long-running soak tests. - Design and execute fault-injection scenarios using process crashes, GDB-driven state manipulation, packet drops, latency injection, corrupted wire data, oversized inputs, and version skew between master and replica nodes. - Identify failure scenarios that conventional positive testing cannot adequately cover. - Investigate complex system defects involving off-by-one errors, out-of-bounds reads, encoder/decoder mismatches, compression and CRC issues, and consistency violations. - Reproduce defects at the source-code level, identify suspect functions, and provide developers with minimal reproducible test cases and detailed root-cause analysis. Test Automation and Framework Development - Build and maintain scalable test frameworks using Python, pytest, custom harnesses, API testing, and end-to-end automation. - Ensure test suites run reliably across shared test infrastructure and CI/CD environments such as GitHub Actions and Jenkins. - Improve test-farm reliability by reducing flaky tests, identifying noisy failures, and strengthening regression coverage. - Develop testing strategies across unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and long-running regression testing. AI-Agent Integration and Testing - Lead the integration of AI agents into existing testing frameworks. - Design agent workflows that can plan and execute tests, interact with tools, inspect logs, traces, metrics, and database state, and generate actionable defect reports. - Develop evaluation frameworks using golden datasets, graders, and regression suites to measure the reliability and accuracy of AI-generated outputs. - Integrate AI agents with pytest, custom testing harnesses, and MCP-style tool servers. - Enable agents to generate negative and edge-case scenarios, maintain flaky tests, and assist with failure triage while retaining human oversight where required. Cross-Functional Collaboration and Technical Leadership - Participate in