AI Evaluator Jobs
AI evaluator jobs: rate, rank and critique AI model answers so the next model improves. Each listing shows the pay, the background it asks for and who can apply.
$125/hr
Typical $125–$125/hr across 20 roles with listed pay
By field
Job listings 20
DataAnnotationSoftware EngineerSoftware & Coding · today WorldwideUp to $150/hrApply
About this role
Run coding sessions with AI models on engineering tasks, then inspect, rate, and red-team their code. The work suits software engineers and developers with experience shipping production code or serious open-source projects.
What you'd do
- Rate and critique AI answers
- Red-team AI models
- Write code
- Review code
Requirements
- No degree required
- Professional or serious open-source experience shipping production code
- Comfort in at least one major stack
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $150/hr
- Eligibility
- Worldwide
- Expertise
- Software & Coding
- Level
- Mid-level
- Posted
- today
DataAnnotationGeneralistGeneralist · today WorldwideUp to $50/hrApply
About this role
Review and rate AI responses for accuracy, helpfulness, and tone, rank outputs using rubrics, and write improved responses when needed. The work suits someone with strong reading comprehension, attention to detail, and clear written English; no technical background or degree is required.
What you'd do
- Rate and critique AI answers
- Write ideal answers
- Write grading rubrics
Requirements
- No prior experience needed
- No degree required
- Strong reading comprehension and attention to detail
- Clear, natural written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $50/hr
- Eligibility
- Worldwide
- Expertise
- Generalist
- Level
- Generalist · no experience needed
- Posted
- today
DataAnnotationData ScientistData & Analytics · today WorldwideUp to $150/hrApply
About this role
Evaluate AI-generated analyses of real datasets, check methods, assumptions, and conclusions, and write improved analyses when needed. Suits people with professional, academic, or independent experience in data analysis, such as data scientists, data analysts, and analytics engineers.
What you'd do
- Rate and critique AI answers
- Red-team AI models
- Write ideal answers
Requirements
- No degree required
- Professional or academic experience doing real data analysis
- Fluency with standard data-analysis tools, including Python, SQL, and notebooks
- Clear written English
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $150/hr
- Eligibility
- Worldwide
- Expertise
- Data & Analytics
- Level
- Mid-level
- Posted
- today
DataAnnotationFinance ExpertFinance & Accounting · today WorldwideUp to $125/hrApply
About this role
You evaluate AI-generated financial models and analyses, trace accounting and modeling logic, and write defensible finance analyses when the models fall short. The work suits people with professional experience in corporate finance, FP&A, or strategic finance and fluency in financial modeling.
What you'd do
- Rate and critique AI answers
- Write ideal answers
Requirements
- No degree required
- Professional experience in corporate finance, FP&A, or strategic finance
- Fluency with three-statement builds, DCF, and the accounting behind them
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Finance & Accounting
- Level
- Mid-level
- Posted
- today
DataAnnotationTax AccountantFinance & Accounting · today WorldwideUp to $125/hrApply
About this role
You draft tax prompts, review AI answers for errors in tax rules and calculations, and write corrected treatments with the reasoning shown. The work suits someone with hands-on corporate, individual, or tax-provision experience who can trace positions to current tax guidance.
What you'd do
- Write prompts and tasks
- Rate and critique AI answers
- Write ideal answers
Requirements
- No degree required
- Hands-on tax experience in corporate, individual, or provision (ASC 740) work
- Comfort tracing a tax position to the code and current guidance
- Clear written English
- Language: EN
Before you start
- Skills assessment
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Finance & Accounting
- Level
- Expert
- Posted
- today
DataAnnotationCorporate AccountantFinance & Accounting · today WorldwideUp to $125/hrApply
About this role
You draft accounting prompts, review AI answers on GAAP, consolidations, and financial close, and write correct treatments with clear working. The work suits someone with hands-on corporate accounting or controllership experience and comfort with GAAP and the close process.
What you'd do
- Rate and critique AI answers
- Write prompts and tasks
- Write ideal answers
Requirements
- No degree required
- Hands-on corporate accounting or controllership experience
- Comfort with GAAP, the close process, and consolidations
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Finance & Accounting
- Level
- Mid-level
- Posted
- today
DataAnnotationLegal ExpertLaw · today WorldwideUp to $125/hrApply
About this role
Draft prompts that test legal reasoning, evaluate model answers for errors, and write corrected legal analysis. The work suits applicants with a JD or equivalent and legal practice experience or a strong academic background.
What you'd do
- Rate and critique AI answers
- Write prompts and tasks
- Write ideal answers
Requirements
- Professional degree (JD, MD, etc.)
- Field: Law
- JD or equivalent
- Practice experience or strong academic background
- Legal research and writing skills across at least one practice area
- Clear written English
- Language: EN
Before you start
- Skills assessment
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Law
- Level
- Expert
- Posted
- today
DataAnnotationAttorneyLaw · today WorldwideUp to $125/hrApply
About this role
You draft legal prompts, evaluate AI-generated legal reasoning, and write corrected analyses when its answers are wrong. The work suits attorneys or other people with a JD or equivalent and legal research and writing experience or a strong academic background.
What you'd do
- Rate and critique AI answers
- Write prompts and tasks
- Write ideal answers
- Fact-check AI output
Requirements
- Professional degree (JD, MD, etc.)
- Field: Law
- JD or equivalent
- Practice experience or strong academic background
- Legal research and writing skills across at least one practice area
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Law
- Level
- Expert
- Posted
- today
DataAnnotationParalegalLaw · today WorldwideUp to $125/hrApply
About this role
The paralegal writes prompts for legal tasks, evaluates AI work for errors, checks it against the record, and writes corrected work product. The work suits someone with hands-on paralegal or legal-assistant experience, especially in litigation, legal research, citation checking, and court procedure.
What you'd do
- Rate and critique AI answers
- Write prompts and tasks
- Write ideal answers
- Fact-check AI output
Requirements
- No degree required
- Hands-on paralegal or legal-assistant experience
- Comfort with legal research, citation checking, and court procedure
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Law
- Level
- Mid-level
- Posted
- today
DataAnnotationHealth/Medical ExpertMedicine & Healthcare · today WorldwideUp to $125/hrApply
About this role
You write clinical prompts, assess AI answers for reasoning errors and safety risks, rate their quality, and provide corrected answers. The work suits someone with clinical or other health-professional experience and clear written English.
What you'd do
- Rate and critique AI answers
- Write prompts and tasks
- Write ideal answers
Requirements
- No degree required
- Clinical or health-professional experience in any specialty
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Medicine & Healthcare
- Level
- Mid-level
- Posted
- today
DataAnnotationRadiologistMedicine & Healthcare · today
+3Up to $125/hrApply
About this role
The role involves reviewing medical imaging studies, writing structured reports and impressions, comparing interpretations, and identifying errors or omissions in AI-generated reports. It suits residency-trained or board-certified radiologists and credentialed or licensed imaging technologists with hands-on imaging experience.
What you'd do
- Rate and critique AI answers
- Write ideal answers
Requirements
- License: Credential or license as a radiologic, ultrasound, or nuclear medicine technologist
- Residency-trained or board-certified radiologist, or credentialed or licensed imaging technologist
- Hands-on experience performing or interpreting studies
- Fluency in English at native or bilingual level
- Clear, well-organized writing
- Language: EN (native)
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Australia, Canada, United Kingdom, Ireland, New Zealand, United States
- Expertise
- Medicine & Healthcare
- Level
- Expert
- Posted
- today
DataAnnotationPhysicianMedicine & Healthcare · today WorldwideUp to $125/hrApply
About this role
The physician writes clinical prompts, evaluates AI answers for safety and quality, and provides evidence-based corrections. The work suits physicians or clinicians with clinical experience and clear written English.
What you'd do
- Write prompts and tasks
- Rate and critique AI answers
- Write ideal answers
Requirements
- Professional degree (JD, MD, etc.)
- Field: Medicine
- Clinical experience
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Medicine & Healthcare
- Level
- Expert
- Posted
- today
DataAnnotationNurseMedicine & Healthcare · today WorldwideUp to $125/hrApply
About this role
The nurse writes prompts about assessment, triage, medication administration, and patient education, then evaluates AI answers and writes corrected answers grounded in bedside practice. The work suits someone with active or recent nursing experience and strong written English.
What you'd do
- Rate and critique AI answers
- Write prompts and tasks
- Write ideal answers
Requirements
- No degree required
- Active or recent nursing experience
- Comfort reasoning through assessment, medication safety, and care protocols
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Medicine & Healthcare
- Level
- Mid-level
- Posted
- today
DataAnnotationQuantitative AnalystFinance & Accounting / Math · today WorldwideUp to $125/hrApply
About this role
You write quantitative finance prompts, assess AI-generated reasoning, and provide corrected analyses when outputs contain errors. The work suits applicants with professional or research experience in quantitative finance, trading, or risk and strong mathematical knowledge.
What you'd do
- Write prompts and tasks
- Rate and critique AI answers
- Write ideal answers
Requirements
- No degree required
- Professional experience in quantitative finance, trading, or risk, or a strong research background
- Fluency with probability, statistics, and stochastic methods
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Finance & Accounting, Math
- Level
- Expert
- Posted
- today
DataAnnotationPortfolio ManagerFinance & Accounting · today WorldwideUp to $125/hrApply
About this role
You evaluate AI-generated investment theses and portfolio allocations, assess their construction and risks, and write improved investment judgments. The work suits people with buy-side portfolio management, hedge fund, or asset-management research experience.
What you'd do
- Rate and critique AI answers
- Write ideal answers
Requirements
- No degree required
- Buy-side experience in portfolio management, hedge fund, or asset-management research
- Fluency with portfolio construction, risk, and security analysis
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Finance & Accounting
- Level
- Expert
- Posted
- today
DataAnnotationHealthcare Operations ManagerMedicine & Healthcare · today WorldwideUp to $125/hrApply
About this role
You create healthcare administration scenarios for AI, evaluate its answers for operational and compliance errors, and write correct answers when needed. The work suits people with hands-on experience in healthcare administration, billing, or practice operations.
What you'd do
- Rate and critique AI answers
- Write prompts and tasks
- Write ideal answers
Requirements
- No degree required
- Hands-on experience in healthcare administration, billing, or practice operations
- Familiarity with common coding and insurance workflows
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Medicine & Healthcare
- Level
- Expert
- Posted
- today
DataAnnotationCybersecurity ExpertSoftware & Coding · today WorldwideUp to $125/hrApply
About this role
You probe AI models for security weaknesses, evaluate their security reasoning, and write secure answers that explain threat models and trade-offs. The work suits applicants with professional experience in security engineering, defensive operations, or offensive security.
What you'd do
- Rate and critique AI answers
- Write ideal answers
- Red-team AI models
Requirements
- No degree required
- Professional experience in security engineering, defensive operations, or offensive work
- Comfort reasoning about authentication, cryptography, injection, and secret handling
- Clear written English
- Language: EN
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Software & Coding
- Level
- Mid-level
- Posted
- today
DataAnnotationFrontend EngineerSoftware & Coding · today WorldwideUp to $150/hrApply
About this role
You test AI-generated frontend components and web features, identify failures in state, accessibility, rendering, and layouts, and write corrected implementations. The work suits frontend engineers, UI engineers, or web developers with professional or serious open-source experience building web UIs.
What you'd do
- Rate and critique AI answers
- Write code
- Write ideal answers
Requirements
- No degree required
- Professional or serious open-source experience building web UIs
- Comfort with a modern framework and CSS
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $150/hr
- Eligibility
- Worldwide
- Expertise
- Software & Coding
- Level
- Mid-level
- Posted
- today
DataAnnotationDevOps EngineerSoftware & Coding / Engineering · today WorldwideUp to $150/hrApply
About this role
You evaluate AI-generated guidance on CI/CD, infrastructure-as-code, containers, and cloud operations, identify operational or security failures, and write corrected approaches with recovery steps. The work suits experienced DevOps, site reliability, or platform engineers with hands-on infrastructure and cloud experience.
What you'd do
- Rate and critique AI answers
- Write ideal answers
- Red-team AI models
Requirements
- No degree required
- Hands-on experience running CI/CD, infrastructure-as-code, or cloud operations
- Comfort with a major cloud and tools such as Terraform, Kubernetes, or Docker
- Clear written English
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $150/hr
- Eligibility
- Worldwide
- Expertise
- Software & Coding, Engineering
- Level
- Mid-level
- Posted
- today
DataAnnotationDesignerDesign & Creative · today WorldwideUp to $125/hrApply
About this role
You evaluate AI-generated design work, identify usability, hierarchy, and accessibility problems, and write critiques explaining how to improve it. The work suits people with professional product, UX, or visual design experience and a portfolio.
What you'd do
- Rate and critique AI answers
- Write ideal answers
Requirements
- No degree required
- Professional experience in product, UX, or visual design
- Portfolio
- Clear written English
- Language: EN
Before you start
- Skills assessment
Full description
- Platform
- DataAnnotation
- Pay rate
- Up to $125/hr
- Eligibility
- Worldwide
- Expertise
- Design & Creative
- Level
- Mid-level
- Posted
- today
Showing 1-20 of 20 positions