What is LLM Robotics? An Introduction to the Field

LLM robotics is the practice of using large language models to translate human instructions into structured, executable commands for physical machines. In this tutorial, I will build a natural-language robot command parser that converts instructions like "pick up the red block and place it in the bin" into a JSON action sequence that a real or simulated arm could consume. This pattern is useful for warehouse automation prototypes, lab assistants, or any project where brittle rule-based parsers break as soon as a user phrases something unexpectedly.

What you'll need

Python 3.10 or newer installed locally. The OpenAI Python SDK, which you can install with pip install openai. An API key from https://portal.oxlo.ai. I am using Oxlo.ai for this project because its request-based pricing means my bill does not scale with prompt length, so I can paste in long sensor logs or detailed robot schemas without watching token counters. That flat cost per request is a significant advantage during early prototyping when prompts are long and unpredictable. Oxlo.ai also serves models with no cold starts, which keeps latency consistent if you eventually move this parser into a live robot loop. You can review current plans at https://oxlo.ai/pricing.

Step 1: Configure the Oxlo.ai client

First, I initialize the OpenAI-compatible client pointing at Oxlo.ai. No custom adapters or wrapper classes are required. I keep the API key in an environment variable so it does not leak into source control.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY"),
)

Step 2: Define the robot command schema

My simulated robot understands four primitives: move, grip, inspect, and speak. Before touching the LLM, I encode this schema as a JSON example inside a Python constant. Giving the model a concrete template reduces hallucinated fields and makes downstream parsing trivial. I deliberately keep coordinates simple. In a production setup you would replace the float triples with frame names from a motion planner, but the LLM does not need to know inverse kinematics. Its job is intent parsing, not trajectory generation.

ROBOT_SCHEMA = """
You must output a single JSON object with this exact structure:
{
  "reasoning": "brief chain-of-thought",
  "commands": [
    {"action": "move", "target": "table", "x": 0.0, "y": 0.0, "z": 0.0},
    {"action": "grip", "target": "red block", "force": 5.0},
    {"action": "speak", "message": "Task complete"}
  ]
}
Allowed actions are: move, grip, inspect, speak.
"""

Step 3: Write the system prompt

The system prompt is the contract between the LLM and the hardware. I include the schema, a strict formatting rule, and a safety reminder. I do not ask for markdown or conversational filler. The model should return only the JSON object. I embed the schema directly into the prompt rather than relying on external function definitions. This keeps the code simple and avoids an extra dependency chain.

SYSTEM_PROMPT = f"""You are a robot command parser.
Your job is to convert a human instruction into structured JSON commands.
{ROBOT_SCHEMA}
Rules:
- Output ONLY valid JSON. No markdown, no explanation outside the JSON.
- If the instruction is ambiguous, use your best reasoning and note it in the "reasoning" field.
- Never generate a move command toward a living thing.
"""

Step 4: Build the command parser

Now I wire the prompt to the LLM. I use Llama 3.3 70B on Oxlo.ai because it follows system instructions precisely and handles structured outputs well. The function parses the content string as JSON and returns a native Python dict, so the rest of my stack can consume it immediately.

import json

def parse_instruction(instruction: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": instruction},
        ],
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 5: Add safety guardrails

Before any JSON reaches a motor controller, I validate it in code. I check that every action is in my allow-list and that no target string contains unsafe keywords. This layer is non-negotiable in physical robotics, and it keeps the LLM from becoming a single point of failure. I keep the validator pure and synchronous so it can run inside a real-time control loop without async overhead.

ALLOWED_ACTIONS = {"move", "grip", "inspect", "speak"}
UNSAFE_KEYWORDS = {"human", "person", "employee"}

def validate_plan(plan: dict):
    if "commands" not in plan:
        raise ValueError("Missing 'commands' key")
    for cmd in plan["commands"]:
        action = cmd.get("action")
        if action not in ALLOWED_ACTIONS:
            raise ValueError(f"Disallowed action: {action}")
        target = str(cmd.get("target", "")).lower()
        if any(k in target for k in UNSAFE_KEYWORDS):
            raise ValueError("Unsafe target detected")
    return True

Run it

I run a few test instructions through the full pipeline. The first is a straightforward pick-and-place request. The second tests ambiguity handling. Both execute in a single request to Oxlo.ai, so the cost is the same regardless of how verbose the system prompt is. If you are iterating on the prompt, that predictability matters. You can send a thousand tokens of scene description and still pay the same flat per-request rate.

if __name__ == "__main__":
    tests = [
        "Pick up the red block from the table and place it in the bin.",
        "Check the conveyor belt for defects and announce the result.",
    ]

    for instruction in tests:
        print(f"\nInstruction: {instruction}")
        try:
            plan = parse_instruction(instruction)
            validate_plan(plan)
            print(json.dumps(plan, indent=2))
        except Exception as e:
            print(f"Error: {e}")

Example output for the first instruction looks like this:

{
  "reasoning": "The user wants to move to the red block on the table, grip it, move to the bin, and release it.",
  "commands": [
    {"action": "move", "target": "red block", "x": 0.5, "y": 0.2, "z": 0.1},
    {"action": "grip", "target": "red block", "force": 5.0},
    {"action": "move", "target": "bin", "x": 1.2, "y": 0.8, "z": 0.1},
    {"action": "grip", "target": "bin", "force": 0.0}
  ]
}

Next steps

This parser is a foundation, not a finished product. Two concrete directions to take it next:

First, integrate with a physics simulator like PyBullet or a ROS 2 node so the JSON commands actually drive a virtual arm. You can publish the command array to a topic and let a motion planner handle inverse kinematics. The JSON schema we defined maps cleanly to ROS action messages.

Second, add vision by upgrading to Kimi K2.6 on Oxlo.ai. Base64-encode a camera frame, append it to the user message, and ask the model to generate targets based on what it sees rather than hardcoded object names. Because Oxlo.ai charges per request rather than per token, adding a detailed image description or a large base64 payload to the prompt does not change your unit cost. That makes vision-to-action loops far more affordable to experiment with than token-based alternatives. You can compare plans at https://oxlo.ai/pricing.