I have been in the human-feedback loop for frontier language models since November 2019, before GPT-3 existed. It began with scalable-oversight experiments at Ought, an alignment research lab. An OpenAI researcher then contracted me to produce human-feedback data, and I did that for five years while the method grew from summary review into full RLHF.I was contracted directly by Long Ouyang, first author of the InstructGPT paper, through Upwork. I was never an OpenAI employee, and I say so because the distinction matters to the people who would hire me. What I offer now is judgement about model output: deciding what good looks like, writing criteria that survive contact with real answers, building prompt and automation pipelines, and finding the places where a model fails without telling you.
What I do
Evaluation and rubric design. I write grading criteria and set up review processes for model output, in technical domains and open-ended ones. The hardest thing to grade was never an explicit error. It was an implied claim, the kind two competent graders can legitimately disagree on.A long answer can be internally inconsistent because of something it implies rather than something it states. Rubrics that only check stated facts miss this entirely. Good rubrics account for that.
Adversarial testing. For years my job included writing code with logic bugs planted in it, subtle enough that a model would miss them, to find where reasoning and error detection break down. The best hiding places are wherever indirection is introduced. As a project's abstractions multiply, so do the places a mistake can live.
Prompt and automation pipelines. The useful question is rarely "can a model do this". It is which parts genuinely need a model, which need plain automation, and which need rethinking. At a digital marketing company I built the ad-copy pipelines while the ad spend they managed grew from roughly $300K to $1M a month, without any new hires.Three systems: a prompt-chain tree for ad copy across Google, Facebook, Instagram, and email, so the team could fork a branch instead of starting over; an internal onboarding assistant tuned on their own material; and a 40-page book draft on restaurant marketing, produced in days.
Agentic development pipelines. Humans decide what the product should be. Agents decompose, implement, test, document, review, and integrate it through an auditable, local-first pipeline. I designed one for my own product and I run it every day. Most teams are trying to build exactly this. How it works.
Evidence
| 6+ years | continuous involvement with frontier language models, from Ought's 2019 experiments to Turing's 2025 training programme |
|---|---|
| 4,384 hours | logged on Upwork across 10 contracts, with a 100% job success score and over $100K earned |
| 1 week | time to promotion to task reviewer at Turing, later flagged as a top performer by the delivery manager |
| 9 papers | AI safety and governance publications I typeset and produced with their authors, including the GPT-4 System Card |
| 1 publication | IEEE conference paper on deep learning for daily solar irradiation, from an M.Sc. in physics |
What it was like
At Ought I played all three roles in a factored-evaluation protocol: honest expert, malicious expert, and judge. Playing the malicious expert was the most fun I have had in any job.One article was an album review written as though the reviewer was at a live concert. My summary claimed the reviewer had literally attended a performance of the album in a bar, and I defended it with slightly cropped quotes from the article itself. The judge believed me. Playing the judge was dreadful. The summary is always suspect, you cannot tell which answers are faithful, and everything has to be re-examined by whatever means you have.
That asymmetry is the oversight problem in miniature, and I have watched it get worse from the inside. Around the arrival of reasoning models, outputs started holding together across thousands of words. I found myself spending serious time catching mistakes in physics I had never studied. I was the oversight, and I was no longer sufficient. That is the problem I want to work on.
Working with me
I take short engagements, from a few days to a few weeks, and ongoing advisory. Typical starting points:
- An evaluation set and rubric for a model-backed feature, with reviewer calibration so two people grade the same way.
- A red-team pass on a prompt, an agent, or a workflow, with written failure cases you can turn into tests.
- A prompt-chain or automation design for a repetitive knowledge task, with a clear line between what needs a model and what does not.
- An agentic development pipeline for a small team, including the review gates.
I work remotely from Ankara (UTC+3) and overlap comfortably with European and US East Coast hours. Turkish and English. I discuss methods and public information, keep client confidences, and decline engagements that conflict with existing ones. Email is the best way to reach me: erolcan.akbaba@gmail.com. A CV is available in English and Turkish, and the longer story is on the background page.