Human in the loop

​

Overview

​

When to use which

• A security question, a one-time code read off a device, or a CAPTCHA your solver can’t handle
• A classification or approval decision—“is this the right invoice?”
• Disambiguation—three accounts matched, which one is it?

• A multi-step identity verification flow that changes every time
• A final submission someone must review on screen before confirming
• An exception path that’s too rare to be worth automating

​

Prerequisites

• A backend service that accepts HTTP POST requests from Intuned, on a publicly accessible HTTPS URL.
• For handOverControlToHuman: the project must run in headful mode ("headful": true). There’s no browser to stream otherwise.

​

Configure your endpoints

{
  "humanInTheLoop": {
    "enabled": true,
    "requestEndpoint": {
      "url": "https://your-app.com/intuned/human-in-the-loop",
      "headers": {
        "Authorization": "Bearer <your-secret>"
      }
    },
    "handoffEndpoint": {
      "url": "https://your-app.com/intuned/control-handoff",
      "headers": {
        "Authorization": "Bearer <your-secret>"
      }
    }
  }
}

​

How it works

​

Asking a person for data

1. Your automation calls humanInTheLoop with a prompt, a data schema, and optional supporting material.
2. Intuned POSTs a request to your requestEndpoint, including a callback URL.
3. Your endpoint returns 2xx to acknowledge it. The automation stays paused—acknowledging is not answering.
4. A person answers in your application. Your backend POSTs the answer to the callback URL.
5. Intuned validates the answer against the schema and resumes the automation, which receives the data as the helper’s return value.

​

Handing over control

1. Your automation calls handOverControlToHuman as its final step.
2. Intuned POSTs a request to your handoffEndpoint, including a stream URL.
3. Your application shows the stream URL to an operator—in an iframe, or a new tab. They see the live browser and have full keyboard and mouse control.
4. The operator finishes the task and selects Complete in the top-right corner of the stream. The stream UI also shows how much time is left.
5. The helper returns and the Run completes.

​

Request payloads

​

human_in_the_loop

{
  "type": "human_in_the_loop",
  "requestId": "hitl_2v9QxLm4Rt",
  "callbackUrl": "https://api.intuned.io/human-in-the-loop/hitl_2v9QxLm4Rt/callback?token=...",
  "prompt": "Which category does this document belong to?",
  "dataSchema": {
    "type": "object",
    "properties": {
      "category": { "type": "string", "enum": ["invoice", "statement", "other"] },
      "confirmed": { "type": "boolean" },
      "note": { "type": "string" }
    },
    "required": ["category", "confirmed"]
  },
  "supportingMaterial": [
    { "type": "image", "url": "https://assets.intuned.io/...signed..." },
    { "type": "text", "text": "Document ID: D-4471" }
  ],
  "metadata": { "documentId": "D-4471", "queue": "billing-review" },
  "timeout": 7200,
  "expiresAt": "2026-10-25T20:14:03Z",
  "workspaceId": "e95cb8d1-f212-4c04-ace1-c0f77e8708c7",
  "project": { "id": "482bf507-5fcc-43ed-9443-d8fff86015c4", "name": "claims-portal" },
  "apiInfo": { "name": "classify", "runId": "656CxOdANRlR5lWUAt_eC", "attempt": 1 }
}

​

control_handoff

{
  "type": "control_handoff",
  "requestId": "handoff_7KpR2nWq8Y",
  "streamUrl": "https://stream.intuned.io/handoff_7KpR2nWq8Y?token=...",
  "prompt": "Complete the identity verification step, then submit the application and select Complete.",
  "supportingMaterial": [],
  "metadata": { "applicantId": "A-8821" },
  "timeout": 1800,
  "expiresAt": "2026-10-25T19:02:11Z",
  "workspaceId": "e95cb8d1-f212-4c04-ace1-c0f77e8708c7",
  "project": { "id": "482bf507-5fcc-43ed-9443-d8fff86015c4", "name": "claims-portal" },
  "apiInfo": { "name": "submit-application", "runId": "9fKd1pQsVz0Lb_aXm", "attempt": 1 }
}

​

The callback contract

curl -X POST "https://api.intuned.io/human-in-the-loop/hitl_2v9QxLm4Rt/callback?token=..." \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "category": "invoice",
      "confirmed": true,
      "note": "Matched to PO-5512"
    }
  }'

​

Implementing the endpoints

interface HumanInTheLoopRequest {
  type: "human_in_the_loop";
  requestId: string;
  callbackUrl: string;
  prompt: string;
  dataSchema: Record<string, unknown>;
  supportingMaterial: Array<
    { type: "text"; text: string } | { type: "image"; url: string }
  >;
  metadata: Record<string, unknown>;
  timeout: number;
  expiresAt: string;
  workspaceId: string;
  project: { id: string; name: string };
  apiInfo: { name: string; runId: string; attempt: number };
}

app.post("/intuned/human-in-the-loop", async (req, res) => {
  if (req.headers.authorization !== `Bearer ${process.env.INTUNED_HITL_SECRET}`) {
    return res.status(401).json({ error: "unauthorized" });
  }

  const request = req.body as HumanInTheLoopRequest;

  // Store the request, including the callback URL, and acknowledge immediately.
  await reviewQueue.enqueue({
    id: request.requestId,
    callbackUrl: request.callbackUrl,
    prompt: request.prompt,
    schema: request.dataSchema,
    material: request.supportingMaterial,
    expiresAt: new Date(request.expiresAt),
    queue: request.metadata.queue,
  });

  res.status(202).json({ received: true });
});

// Later, when a person submits their answer in your UI:
async function submitAnswer(requestId: string, data: unknown) {
  const { callbackUrl } = await reviewQueue.get(requestId);

  const response = await fetch(callbackUrl, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ data }),
  });

  if (response.status === 400) {
    const { error } = await response.json();
    throw new Error(`Answer rejected by schema validation: ${error}`);
  }
  if (!response.ok) {
    throw new Error(`Callback failed with ${response.status}`);
  }
}

import os
from datetime import datetime

import requests
from flask import jsonify, request


@app.route("/intuned/human-in-the-loop", methods=["POST"])
def intuned_human_in_the_loop():
    expected = f"Bearer {os.environ['INTUNED_HITL_SECRET']}"
    if request.headers.get("Authorization") != expected:
        return jsonify({"error": "unauthorized"}), 401

    payload = request.get_json()

    # Store the request, including the callback URL, and acknowledge immediately.
    review_queue.enqueue(
        id=payload["requestId"],
        callback_url=payload["callbackUrl"],
        prompt=payload["prompt"],
        schema=payload["dataSchema"],
        material=payload["supportingMaterial"],
        expires_at=datetime.fromisoformat(payload["expiresAt"].replace("Z", "+00:00")),
        queue=payload["metadata"].get("queue"),
    )

    return jsonify({"received": True}), 202


# Later, when a person submits their answer in your UI:
def submit_answer(request_id: str, data: dict):
    entry = review_queue.get(request_id)

    response = requests.post(entry.callback_url, json={"data": data}, timeout=10)

    if response.status_code == 400:
        raise ValueError(f"Answer rejected by schema validation: {response.json()}")
    response.raise_for_status()

​

Timeouts and retries

• timeout is per call, in seconds, defaulting to one hour. Set it from how long a person realistically needs—a queued review might be hours, a live handoff minutes.
• The attempt timeout doesn’t apply to the wait. requestTimeout keeps governing your automation’s own execution time.
• A timeout fails the attempt. If the Run has retries left, the next attempt runs from the top and issues a new request—the person gets asked again. Wrap the call and throw NonRetryableError if asking twice would be wrong.
• Requests are tied to an attempt. The attempt field in apiInfo lets you detect and collapse duplicates in your queue.

​

Security

• Authenticate inbound requests. Intuned sends the headers you configure on every request. Verify them before acting, and reject anything else.
• Treat callbackUrl and streamUrl as secrets. Both carry a bearer token. Anyone with the callback URL can answer on your behalf; anyone with the stream URL can drive the browser. Don’t log them, don’t put them in URLs you render to untrusted users, and scope them to the operator who’s handling the request.
• Both URLs expire with the request and are single-use.
• A handoff exposes whatever is on screen. The operator sees the live session, including any data already loaded. Only hand control to people authorized to see it.
• Pause tracing around a handoff if the person might type a password or a one-time code. The trace keeps recording while they work. See Pause tracing.

​

Best practices

• Write prompts for a person who lacks your context. “Pick the account ending 4471” beats “disambiguate account”.
• Attach a screenshot. A page.screenshot() as supporting material is usually the difference between a fast answer and a wrong one.
• Keep schemas small. Ask for the decision, not a transcription of the page. Fewer fields means fewer rejected callbacks.
• Use metadata for routing, so your endpoint doesn’t have to parse prompts to decide who should handle a request.
• Make your endpoint idempotent on requestId. Retries and attempt-level retries both produce repeat deliveries.
• Alert on requests nearing expiresAt, so a forgotten review doesn’t silently fail a Run.

​

Limitations

• You build the operator UI. Intuned doesn’t provide a review queue or an inbox. The endpoint contract is the integration point.
• Your endpoint must be publicly reachable over HTTPS. There’s no polling alternative during preview.
• Handoff requires headful mode, which uses more resources than headless—see the machine size guidance in Intuned.json.
• A handoff has no return value. The person’s work shows up as page state, not as data. Call it last.
• Waiting occupies a machine. A Run paused on a person still holds its slot against your project’s maxCount, so a queue of waiting Runs can block new ones. Size replication with that in mind.
• One request at a time per automation. Both helpers block until they resolve; concurrent requests from a single Run aren’t supported.

​

FAQs

​

Related resources