Validate structured output

Learn it

Words to know

JSON

A text format for data: objects in { }, lists in [ ], strings, numbers, true, false and null.

structured output

A model reply in a fixed data format, such as JSON with agreed fields, so that a program can read it.

schema

The agreed shape of a piece of data: which fields it has, their types, and which values are allowed.

How it works

1. Parse the text as JSON. If parsing fails, the reply is invalid.
2. Check the shape: the value must be an object with exactly two keys. null, a list, a number, a string, true and false are valid JSON too, and all of them are rejected.
3. Check each value: category is one of billing, technical and other; confidence is a number from 0 to 1, both included.
4. Answer true only if every check passed. Never raise an error: a bad reply is an answer of false.

Worked example

Input
text = '{"category":"billing","confidence":"0.8"}'

Output
false

Four checks pass and the last one fails, so the reply is rejected. A reply is valid only when every check passes.

Why it matters

This is the gate between a model and the rest of your program. The second roadmap milestone builds a ticket classifier around this check, and tests it with saved replies, valid and broken.

Your turn

Decide whether a model's reply is a valid support-ticket classification.

text

the model's reply, as a string; it may not be JSON at all

Returns

True if the reply is a JSON object with exactly the keys category and confidence, an allowed category, and a confidence that is a number from 0 to 1, both included. Otherwise false; in Python these are True and False. It must never raise an error.

Input
text = '{"category":"billing","confidence":0.8}'

Output
true

Input
text = '{"category":"billing","confidence":"0.8"}'

Output
false

Input
text = '{"category":"admin","confidence":0.8}'

Output
false

Input
text = "not json"

Output
false

The checks also try

• a confidence of exactly 0
• a confidence of exactly 1
• a confidence above 1
• a confidence below 0
• an extra key
• the JSON value null
• a list instead of an object
• true as the confidence
• a bare number
• a list as the category
• a list that holds a category and a number
• two keys, one with the wrong name
• null as the confidence
• the two keys in the other order
• a category with a capital letter

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Parsing can fail, so it has to happen inside the language's block for catching errors. If it fails, the answer is false.

2. Hint 2: The key step

   Check the shape before the values. A parsed reply can be null, a list, a number or a string, and none of those has the two keys to look at.

3. Hint 3: The outline

   Parse inside a try block, and return false if that raises an error. Return false unless the value is an object, not null and not a list, with exactly two keys. Return false unless category is one of the three names. Return true only if confidence has the type number and lies from 0 to 1, both included.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Parse the text; a reply that is not JSON is invalid. Parsing raises an error on broken text. Catching it turns the error into the answer false, as the task requires.
2. Only an object with exactly two keys can be right. null, lists, numbers and strings are all valid JSON, and none of them is a classification. An extra key means the model added something you did not ask for.
3. The category must be one of the three names. A model can invent a label such as admin. The code after this point only knows how to handle the three names you defined.
4. The confidence must be a number from 0 to 1. The text 0.8 in quotes and the value true are not numbers. Checking the type first keeps the range check from comparing the wrong kind of value.

The same numbered steps are the comments in the solution.

function solve(text) {
  // 1. Parse the text; a reply that is not JSON is invalid
  let reply;
  try {
    reply = JSON.parse(text);
  } catch {
    return false;
  }
  // 2. Only an object with exactly two keys can be right
  if (reply === null || typeof reply !== "object") return false;
  if (Array.isArray(reply) || Object.keys(reply).length !== 2) return false;
  // 3. The category must be one of the three names
  const categories = ["billing", "technical", "other"];
  if (!categories.includes(reply.category)) return false;
  // 4. The confidence must be a number from 0 to 1
  const confidence = reply.confidence;
  if (typeof confidence !== "number") return false;
  return confidence >= 0 && confidence <= 1;
}

import json


def solve(text):
    # 1. Parse the text; a reply that is not JSON is invalid
    try:
        reply = json.loads(text)
    except ValueError:
        return False
    # 2. Only an object with exactly two keys can be right
    if not isinstance(reply, dict) or len(reply) != 2:
        return False
    # 3. The category must be one of the three names
    if reply.get("category") not in ("billing", "technical", "other"):
        return False
    # 4. The confidence must be a number from 0 to 1
    confidence = reply.get("confidence")
    if type(confidence) not in (int, float):
        return False
    return 0 <= confidence <= 1

Watch out

• Valid JSON is not a valid reply. null, [] and 5 all parse without an error, so parsing alone proves nothing about the shape.
• Do not convert values to make them fit. Turning the text 0.8 into a number hides the fact that the model broke the agreement.
• In JavaScript: In JavaScript typeof null is object, and a list is an object too. Rule out both before you look at the keys.
• In Python: In Python True counts as the number 1: isinstance(True, int) is true. Compare the exact type, so a boolean is not accepted as a confidence.

Check yourself