AI Test Architect

Job Description

Job Ttitle: AI Test Architect

Company: Qualitest


Description: Are you interested in working with the World’s leading AI-powered Quality Engineering Company? Ready to advance your career, team up with global thought leaders across industries and make a difference every day? Join us at Qualitest!We are looking for an AI Test Architect to join our growing team in United States!Role – AI Test ArchitectLocation – Remote (US)Job Description:– Architect and build scalable test automation frameworks for generative AI models, leveraging tools like DeepEval, Langsmith, PyTest, or custom Python scripts.– Implement automated evaluation workflows for model validation, including functionality, accuracy, performance, and reliability testing.– Design continuous testing pipelines integrated with CI/CD for model deployments.– Proven expertise in test automation frameworks for AI/ML systems, with a focus on generative models and NLP applications.– Hands-on experience with DeepEval, Langsmith, PyTest, LangChain, or similar evaluation libraries.– Proficiency in programming languages: Python (primary), JavaScript, or TypeScript.– Solid understanding of AI evaluation metrics (BLEU, ROUGE, embedding-based metrics) and adversarial testing techniques.Role Overview:The Automation Architect will design and implement a robust automation framework for testing generative AI models, chatbots, and related AI-powered systems. The role involves optimizing testing pipelines for large-scale, high-throughput models, ensuring quality, scalability, and performance.Roles and Responsibilities:

• Architect and build scalable test automation frameworks for generative AI models, leveraging tools like DeepEval, PyTest, or custom Python scripts.

• Implement automated evaluation workflows for model validation, including functionality, accuracy, performance, and reliability testing.

• Design continuous testing pipelines integrated with CI/CD for model deployments.

• Develop modular frameworks to automate evaluation of NLP-based outputs, including coherence, relevance, toxicity, bias, and safety checks.

• Optimize test execution for large-scale LLM deployments, minimizing latency and maximizing throughput.

• Automate performance and load testing for high-token transactions and API-driven AI endpoints.

• Leverage vector database testing to ensure quality of data retrieval and seamless model integration.

• Introduce monitoring mechanisms to validate models post-deployment and detect model drift.

• Collaborate with DevOps and AI/ML teams to integrate AI model testing within MLOps workflows.

• Drive adversarial and stress testing automation to assess AI robustness and edge-case handling.

Skills and Qualifications:

• Proven expertise in test automation frameworks for AI/ML systems, with a focus on generative models and NLP applications.

• Hands-on experience with DeepEval, PyTest, LangChain, or similar evaluation libraries.

• Proficiency in programming languages: Python, JavaScript, or TypeScript.

• Knowledge of vector databases (e.g., Pinecone, Milvus) and their role in AI model pipelines.

• Experience with CI/CD pipelines, version control (Git), and MLOps platforms.

• Familiarity with tools for performance testing, such as JMeter, Locust, or custom scripts.

• Solid understanding of AI evaluation metrics (BLEU, ROUGE, embedding-based metrics) and adversarial testing techniques.

• Strong architectural skills to build reusable, scalable automation solutions.

Why QualiTest?

• Be a part of a company who strives to support for diversity and inclusion in the workplace – we are one, we are many at Qualitest. Celebrate culture, share knowledge with engineers from around the globe, and inspire each other through our differences.
• Local and global opportunities – we offer you internal rotation and international mobility opportunities to grow your career.
• Clear view of your career and progression with the company – Qualitest is growing massively and giving you the opportunity to grow with us.
• Work hard and play harder with our flexible and casual culture. Take a break from work and join an employee event, or enjoy the amenities and games provided from one of our Employees Centers.
• Save your earnings and prepare for your future by enrolling in our 401k plan where Qualitest will match your contributions accelerating your savings plan.
• Take care of health with enrollment into one of our competitive healthcare benefits. Qualitest will match towards your HSA if you choose to participate.
• Never stop experimenting and learning with Qualitest Tech academy: 3000+ training courses, mentorship programs, technical tribes, sponsored certifications, leadership programs and much more
• Stay active and get rewarded with our Corporate Wellness Program. We pay your Gym membership and giving you opportunities to Earn additional vacation times for attendance the gym!
• Competitive Salary of $170k-180k per year
• Earn bonuses via our Client Referral and Employee Referral Program’s. Refer and earn – tap your network for net-worth.
• Planning a vacation? Looking for car insurance? Get access to Qualitest Employee Perks for discounts on anything from travel to electronics. With so many offerings the savings are endless!

Intrigued to find more about us?

• Visit our website at

If you like what you have read, send us your resume and let’s start talking!

Salary: $170000 – 180000 per year

Location: Santa Clara, CA

Date: Thu, 26 Dec 2024 05:57:29 GMT


Read More…