QA Engineer (AI & Feature Testing)

Accountabilities

Plan and execute comprehensive testing: Design, develop, and execute test plans, test cases, and test scripts for traditional software features and AI-powered components across web, mobile, and API environments.
Validate AI-driven functionality: Test AI and LLM-integrated features for accuracy, relevance, consistency, robustness, and safety, identifying hallucinations, unexpected responses, and failures in edge cases.
Apply appropriate testing methodologies: Determine when to use deterministic pass/fail testing and when to apply evaluation-based approaches for probabilistic AI outputs, ensuring test strategies reflect the expected behavior of each feature.
Evaluate AI model performance: Assess AI-generated outputs using appropriate evaluation metrics, benchmark datasets, semantic assessments, and human-in-the-loop review processes.
Translate requirements into test scenarios: Analyze business requirements, user journeys, process documentation, and AI behavior specifications to develop comprehensive and effective testing coverage.
Conduct manual feature testing: Execute functional tests across web and mobile applications, verify expected behavior, validate fixes, and ensure new releases meet quality expectations.
Perform regression and integration testing: Validate interactions between application components and APIs while conducting regression, performance/load, and user acceptance testing (UAT).
Identify and document defects: Create clear, detailed bug reports that include reproducible steps, expected and actual results, supporting evidence, and relevant technical information.
Investigate technical issues: Use browser developer tools, application logs, and debugging utilities to investigate failures, trace unexpected behavior, and support efficient root-cause analysis.
Support mobile quality assurance: Validate mobile application behavior across relevant iOS and Android environments, considering platform-specific functionality and potential compatibility issues.
Improve QA practices: Contribute to testing standards, methodologies, and best practices for emerging technologies, including AI frameworks and LLM-based applications.
Maintain testing documentation: Use Jira, Confluence, and test management systems to track defects, organize test cases, document results, and maintain clear project records.
Collaborate across teams: Partner with engineers, functional specialists, product stakeholders, and IT teams to resolve issues, align testing priorities, and ensure quality throughout the development lifecycle.
Develop subject-matter expertise: Build a strong understanding of core systems, application interfaces, business processes, and reporting tools to strengthen test coverage and improve the reliability of delivered solutions.
Promote responsible AI quality: Help ensure AI-powered features meet appropriate standards for reliability, robustness, safety, and user trust.
Requirements
Professional experience: 5–7 years of hands-on software testing experience across web applications, mobile platforms, and APIs.
Playwright expertise: Proven production experience using Playwright for software testing. This is a primary requirement for the position.
AI and LLM testing: Practical experience testing AI-powered products or LLM-integrated features, including evaluating accuracy, relevance, robustness, adversarial prompt handling, edge cases, and safety or toxicity risks.
Testing methodology: Clear understanding of the differences between deterministic testing, which relies on defined inputs and expected outputs, and evaluation-based testing for probabilistic AI systems.
AI evaluation techniques: Familiarity with human-in-the-loop evaluations, scoring metrics, benchmark datasets, semantic evaluation, and other approaches for assessing AI-generated responses.
Software testing fundamentals: Demonstrated experience with functional, integration, regression, performance/load, and user acceptance testing.
Web and mobile development knowledge: Strong understanding of application development lifecycles, including web technologies and platform-specific considerations for iOS and Android.
Debugging and diagnostics: Hands-on experience with browser developer tools, application logs, and technical investigation techniques for identifying and resolving defects.
Test management and documentation: Advanced proficiency in Jira and Confluence, along with experience using test management systems to organize test cases, execution results, and defect tracking.
Analytical thinking: Strong attention to detail and the ability to investigate complex issues, recognize unexpected patterns, and distinguish between software defects and acceptable variations in AI-generated outputs.
Communication skills: Ability to document technical findings clearly, communicate quality risks effectively, and collaborate with cross-functional stakeholders.
Teamwork and adaptability: Comfortable working in a distributed, international environment, coordinating with engineering and functional teams across different locations.
Preferred technical experience: Familiarity with HTTP proxy and debugging tools such as Charles Proxy or Fiddler, as well as Xcode logs and Android Debug Bridge (ADB) logs.
Preferred industry experience: Previous experience in financial services, fintech, or personal finance technology is an advantage.
Preferred education: A bachelor's degree in Computer Science or an equivalent technical qualification.
Quality and ethics mindset: Commitment to continuous improvement, responsible AI practices, and delivering reliable, secure, and user-focused software.
Benefits
100% remote work: Work remotely from Brazil while collaborating with an international technology team.
International collaboration: Work with colleagues across countries and cultures, contributing to projects with global reach.
Meaningful AI projects: Help build and validate real-world AI-powered solutions and contribute to the evolution of AI-enabled products.
Professional development: Strengthen your expertise in software quality assurance, advanced testing methodologies, and AI/LLM evaluation.
Continuous learning: Join an environment that encourages technical development, knowledge sharing, and experimentation with emerging technologies.
Inclusive workplace: Contribute to a collaborative culture that values diversity, equal opportunity, and different perspectives.
Responsible AI focus: Participate in work that emphasizes ethical AI standards, product reliability, and user trust.
Career growth opportunities: Expand your technical capabilities by working on challenging projects spanning software engineering, AI, and enterprise technology.