What should an AI hiring agent log?

Roku is a recruitment platform with an AI screening layer. The important design constraint is that the AI is advisory: people make the hiring decisions. That changes what the system needs to record. A score alone is not enough; the useful artifact is the path that produced it.

The gap between a demo and a review

A demo can show a candidate being scored and a recommendation appearing. A reviewer needs to know which agent ran, what input it used, what it returned, whether it failed, and how long it took. Without that, a surprising recommendation is impossible to debug and impossible to defend.

Log the tool call, not just the answer

Roku stores tool execution logs with the tool name, input, result, status, and duration. Each log is attached to the chat message that triggered it and to the specific agent that ran: resume parser, candidate scorer, or hiring recommendation.

tool_execution_logs
  chat_message_id
  loggable_type / loggable_id
  tool_name
  input (json)
  result (json)
  status
  duration_ms

The polymorphic link is deliberate. The same log table serves three agents without three near-identical tables. A stable morph map keeps the type names readable and avoids storing full class names in the database.

Scoping and deletion are part of the model

Recruitment records are isolated by company, and logs hang off the chat session that produced them, so authorization can follow the same ownership path. Deletion rules matter too: if a candidate or an agent is removed, the logs should not become orphaned records that nobody can interpret. Cascading deletes and model hooks handle the normal paths; the database constraints catch the rest.

What I would add next

• A prompt version and model identifier on every run, so a change in behavior is traceable to a change in configuration.
• A redacted input snapshot, because the original input may contain personal data that should not live in logs forever.
• A review workflow that records the human decision separately from the AI recommendation, so the advisory boundary is visible in the data.

None of this makes the model more accurate. It makes the model usable in a process where a person is accountable. That is the standard an AI feature in a hiring product should meet.