AI Operations Specialist

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for an AI Operations Specialist based in the United States.

As an AI Operations Specialist, you will help business and product teams transform manual review and quality assurance processes into efficient, reliable AI-powered workflows. You will identify opportunities for automation, map existing processes, and develop solutions that combine AI capabilities with human oversight. Working closely with product managers, business leaders, engineers, legal partners, and subject matter experts, you will establish measurable quality standards and validate AI performance before deployment. You will also help determine whether AI-powered features are ready for launch by evaluating accuracy, consistency, and reliability. This hands-on role offers the opportunity to take ownership of projects from initial design through implementation, monitoring, and continuous improvement. Join a dynamic, AI-driven environment where experimentation, operational excellence, and measurable business impact are central to success.

• Identify, assess, and prioritize AI automation opportunities based on business impact, operational efficiency, feasibility, and expected value.
• Collaborate with process owners to map existing workflows, identify bottlenecks, and determine which tasks can be automated with AI and which require human intervention.
• Design, build, and implement AI-powered workflows, including prompt development, pipeline configuration, quality controls, and performance monitoring.
• Establish and maintain quality standards for AI-driven processes, ensuring outputs remain accurate, consistent, and aligned with business requirements.
• Develop evaluation datasets and test cases to measure AI performance using metrics such as accuracy, precision, recall, and sampling-based quality assessments.
• Evaluate and calibrate large language model (LLM) judges against human reviewer decisions to improve the reliability and consistency of automated evaluations.
• Partner with product teams to define launch-readiness criteria for AI features, generate performance evidence, and support go/no-go decisions.
• Translate complex or ambiguous policies into clear, actionable evaluation criteria that reviewers can apply consistently across different scenarios.
• Work with subject matter experts to develop labeling guidelines and produce high-quality, human-verified labels using real-world data.
• Coordinate large-scale data labeling and review operations with external vendor teams, validating their work through agreement metrics, sampling, and quality checks.
• Monitor deployed AI workflows for performance degradation, quality drift, and emerging issues, implementing corrective actions when necessary.
• Take ownership of assigned workstreams from initial planning and setup through implementation, quality validation, and ongoing monitoring.
• Document workflows, evaluation methodologies, lessons learned, and best practices to make solutions reusable and accelerate future deployments.
• Stay informed about evolving AI models, tools, evaluation techniques, and vendor capabilities, continuously improving existing solutions and retiring approaches that are no longer effective.
• Communicate findings, performance metrics, and recommendations clearly to business stakeholders, product managers, and engineering teams.

Requirements:

• Between 5 and 7 years of professional experience in operations, program management, analytics, consulting, applied AI, or a related field, with demonstrated accountability for delivering measurable results.
• Hands-on experience using AI tools regularly and building an AI-powered workflow, automation, or analytical solution that has delivered tangible business impact.
• Strong understanding of performance measurement, including metrics such as accuracy, precision, recall, and sampling methods.
• Demonstrated ability to transform ambiguous, complex, or inefficient processes into structured workflows with clear criteria, measurable outcomes, and repeatable quality standards.
• Experience in quality assurance, auditing, review operations, guideline development, or comparable quality management activities.
• Strong written and verbal communication skills, including the ability to document processes clearly and collaborate effectively with business leaders, product managers, engineers, and other stakeholders.
• Ability to work independently, manage competing priorities, take ownership of projects, and make sound decisions with limited supervision.
• Strong analytical thinking and problem-solving skills, with a practical, results-oriented approach to evaluating AI performance and improving operational processes.
• Willingness to learn new AI evaluation methods, tools, and technologies as the scope of responsibilities evolves.
• A technical degree or formal technical background is not required. Previous experience designing AI evaluations or calibrating LLM judges is also not mandatory, as training will be provided.
• Preferred: Practical experience with SQL, Python, APIs, or other tools used to analyze data and build automated workflows.
• Preferred: Experience in trust and safety, content moderation, data annotation, labeling operations, or other human-in-the-loop processes at scale.
• Preferred: Experience with model fine-tuning, AI evaluation frameworks, or calibrating LLM-based judges.
• Preferred: Experience in a two-sided marketplace, travel industry, or another operations-intensive business environment.
• Preferred: Experience establishing intake processes, prioritization frameworks, or operational procedures for a newly formed team.
• Work authorization: Must be authorized to work in the United States without requiring employer sponsorship.
• Availability: Able to commit to a 12-month, full-time contract beginning November 23, 2026, with a schedule of 40 hours per week, typically eight hours per day over five days.

Benefits:

• Hourly compensation: $65–$75 per hour, depending on qualifications and experience.
• Additional benefits: Benefits are offered as part of the W-2 contract engagement; specific coverage and eligibility details should be confirmed during the hiring process.
• Remote work: Fully remote opportunity for candidates located within the United States.
• Full-time schedule: 40 hours per week, providing a consistent work schedule throughout the contract.
• Contract duration: Initial 12-month engagement, offering the opportunity to contribute to meaningful AI operations initiatives.
• Hands-on AI experience: Opportunities to design, implement, evaluate, and optimize real-world AI workflows with measurable business impact.
• Cross-functional collaboration: Work alongside business, product, engineering, legal, and subject matter experts to deliver practical AI solutions.
• Professional development: Training and opportunities to develop expertise in AI evaluation, quality assurance, workflow automation, and large language model performance assessment.
• End-to-end ownership: The opportunity to manage AI operations projects from initial discovery and design through deployment, quality monitoring, and continuous improvement.
• Innovation-focused environment: Exposure to evolving AI technologies, operational challenges, and scalable automation strategies.