Role Overview
Mercor is seeking SWE Experts to support the design of evaluation-ready workflows for advanced AI systems. This engagement focuses on translating ambiguous requirements into structured, repeatable artifacts that can be tested automatically. You’ll produce clearly specified deliverables (documentation + scripts) that enable consistent assessment of agent performance across scenarios. Work is contract-based, outcome-oriented, and optimized for reproducibility and clear acceptance criteria.
Key Responsibilities
Convert high-level objectives into tightly scoped, testable deliverables with clear inputs/outputs and measurable success criteria.
Create structured documentation that defines expected behavior, constraints, and edge cases in a way other evaluators can reuse.
Build lightweight automation scripts to support evaluation flows (e.g., generating required artifacts, validating outputs, enforcing format rules).
Write deterministic Python verifier scripts that check completion via final state or output validation (files, directories, content assertions).
Design prompts/tasks that reliably elicit the target workflow behavior while avoiding leakage of internal instructions or implementation details.
Implement robust error handling and actionable failure messages in verification tooling.
Develop plausible but ineffective “baseline” or “distractor” approaches to confirm evaluation discrimination (i.e., the solution must use the intended approach).
Maintain clean artifact hygiene: versionable structure, consistent naming, minimal ambiguity, and reproducible execution.
Ideal Qualifications
Strong Python skills (file system operations, parsing, validation, test-style assertions, deterministic execution).
Experience with evaluation harnesses, automated grading, or QA-style verification (unit/integration test mindset).
Familiarity with prompt design and LLM evaluation methodologies (closed-ended tasks, leakage avoidance, reliability testing).
Comfort with structured specs and documentation conventions (Markdown, YAML frontmatter patterns, well-scoped requirements).
Working knowledge of common developer tooling: Git, CLI workflows, virtual environments, dependency management.
Bonus: embeddings/similarity concepts (e.g., cosine similarity) for “looks relevant but fails” negative-control design.
Ability to communicate clearly and keep scope controlled without relying on domain-specific context.
More About the Opportunity
Deliverables are primarily documentation + scripts intended to support automated evaluation and consistent replay.
Emphasis on: determinism, reproducibility, closed-ended outcomes, and strong verifier reliability.
Tasks and validators should be resilient to superficial shortcuts and confirm the intended workflow is actually used.
Work can include designing negative controls (distractors) that appear credible while failing for principled reasons.
Time-sensitive elements should be explicitly date-bounded where applicable.