Software Engineering Evaluator (contract)
Software EngineeringRemote (US, Canada, and UK only)Contract2–3 yrs$100–$150 / hour
Overview
MarketerHire is seeking a Software Engineering Evaluator for a remote, hourly contract supporting the assessment of AI systems that write, debug, and review production code. The role requires 2–3 years of professional software engineering experience, strong command of at least one listed programming language, sound software engineering fundamentals, code review experience, excellent written English, and the ability to evaluate complex code with independent, evidence-based judgment.
Responsibilities
Key Responsibilities
- Design task-specific grading criteria for feature implementations, bug fixes, refactors, tests, and code reviews.
- Evaluate AI-generated code for correctness, including failures that pass the code's own tests.
- Assess edge-case handling, security, performance, and readability in coding deliverables.
- Write clear, evidence-based justifications for every evaluation score.
- Apply consistent judgment across software engineering evaluation tasks.
- Review code that may appear correct and run without failing while identifying subtle defects.
- Reason through ambiguity and distinguish clearly between known facts and inferences during evaluations.
- Iterate on evaluation work based on feedback from senior reviewers.
Requirements
Required Qualifications
- 2–3 years of professional software engineering experience shipping production code.
- Strong command of at least one of Python, JavaScript/TypeScript, Java, Go, C++, C#, or a comparable language.
- Solid grounding in data structures, algorithms, testing, debugging, and version control.
- Experience reviewing other engineers' code.
- Excellent written English.
- Ability to follow complex instructions, reason through ambiguity, and spot subtle errors in code that looks correct and runs without failing.
- Ability to clearly separate what you know from what you are inferring.
- Ability to work independently, learn new tools quickly, and take feedback well.
- Based in and working from the US, Canada, or UK.
- Available for 30–40+ hours per week for the full 12 weeks.
Benefits
Benefits and Perks
- $100–$150 per hour, paid weekly via Stripe or Wise.
- Remote work on your own schedule within the 30–40+ hour weekly engagement.
- Initial 12-week contract with the possibility that projects may be extended based on needs and performance.
- Opportunity to evaluate AI systems for leading AI research organizations.
- Reasonable accommodations available upon request.