We are going to build a support ticket triage agent that classifies requests, scores its own confidence, and drafts a response. It is meant for teams where a misrouted bug or missed security report is expensive. The pipeline uses multiple models on Oxlo.ai to separate reasoning, verification, and generation, so each step runs on the right tool for the job.
What you'll need
- Python 3.10 or newer
pip install openai pydantic- An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Bootstrap the client and schema
I start every accuracy project with a rigid schema. Unstructured text is the enemy of reliability. When a model outputs freeform markdown, you are forced to parse meaning with regex or hope, and both break in production. Pydantic gives us runtime type enforcement, and Oxlo.ai's JSON mode guarantees the model returns parseable objects rather than prose wrapped in apologies. I pick llama-3.3-70b for the first pass because it is responsive, handles long context windows well, and follows system instructions tightly. Because Oxlo.ai loads popular models with no cold starts, the first request after a deploy returns immediately, which matters when you are validating a new prompt in a tight loop.
from openai import OpenAI
from pydantic import BaseModel, Field
import json
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
class TicketAnalysis(BaseModel):
reasoning: str = Field(description="Step-by-step thinking")
category: str = Field(description="One of: bug, billing, feature_request, security")
priority: str = Field(description="One of: low, medium, high, critical")
confidence: int = Field(description="Integer 1-10")
draft_reply: str = Field(description="Short initial response")
Step 2: Lock in the system prompt
The system prompt is where accuracy is won or lost. I force chain-of-thought reasoning so the model must lay out evidence before it commits to a category. This single change reduces hallucination more than any temperature tweak. I also hardcode guardrails: critical priority is reserved for data loss or complete outage, and any mention of unauthorized access triggers a security classification. Forcing the model to write reasoning into a dedicated JSON field makes its logic inspectable. Treat this prompt like config, not code, and version it.
SYSTEM_PROMPT = """You are a senior support engineer triaging incoming tickets.
Rules:
- Write your reasoning first. Consider keywords, user sentiment, and blast radius.
- Category must be exactly one of: bug, billing, feature_request, security.
- Priority critical is reserved for: data loss, security breach, or complete outage affecting all users.
- If the user mentions unauthorized access, breach, or leaked credentials, category is security and priority is at least high.
- If confidence is below 7, flag it honestly. Do not invent confidence to please the user.
Respond ONLY as valid JSON matching the required schema."""Step 3: Build the structured classifier
With the schema and prompt in place, the classifier becomes a single JSON mode call at low temperature. Temperature 0.1 keeps the model conservative and reduces creative reinterpretation of the guardrails. Oxlo.ai supports the same OpenAI SDK response_format flag, so switching providers requires no code changes. We validate the output against our Pydantic model immediately, so any malformed response or type mismatch crashes loudly during development instead of silently corrupting downstream logic.
def classify_ticket(ticket_body: str) -> TicketAnalysis:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": ticket_body},
],
response_format={"type": "json_object"},
temperature=0.1,
)
return TicketAnalysis(**json.loads(response.choices[0].message.content))
Step 4: Add a verification layer
One pass is never enough for high-stakes edge cases. When the first pass returns confidence below 8, or when the category is security, I run a second opinion through deepseek-r1-671b. This is the critic pattern: a dedicated reasoning model audits the work of the first. Its MoE architecture handles nuanced policy reasoning, and a separate critic step catches overconfidence or misinterpretation that a single model might not surface. On a token-based provider, adding this second call with a long audit prompt and a lengthy ticket thread would balloon costs unpredictably. On Oxlo.ai, the per-request pricing means the extra verification costs the same whether the ticket history is ten words or ten thousand. That predictability lets you build safer systems without budgeting defensively.
def verify_ticket(ticket_body: str, first_pass: TicketAnalysis) -> TicketAnalysis:
if first_pass.confidence >= 8 and first_pass.category != "security":
return first_pass
audit_prompt = (
"You are a principal engineer auditing a support triage decision.\n"
f"Original reasoning: {first_pass.reasoning}\n"
f"Proposed category: {first_pass.category}\n"
f"Proposed priority: {first_pass.priority}\n"
"Re-evaluate the ticket and output corrected JSON if you disagree. "
"Otherwise echo the original fields but set confidence to 10."
)
response = client.chat.completions.create(
model="deepseek-r1-671b",
messages=[
{"role": "system", "content": audit_prompt},
{"role": "user", "content": ticket_body},
],
response_format={"type": "json_object"},
temperature=0.1,
)
return TicketAnalysis(**json.loads(response.choices[0].message.content))
Step 5: Add escalation guardrails
Even good reasoning needs a safety rail. I block auto-replies when the final confidence is below 7 or when priority is critical. A high-accuracy system is not one that never makes mistakes. It is one that knows when it is uncertain and escalates to a human. Confidence calibration is difficult with current models, so I treat the score as a rough gradient rather than a probability. Anything below 7 means the evidence was ambiguous, and ambiguous tickets go to the queue. Critical priority always goes to a human because the cost of a wrong auto-reply, especially in security or outage scenarios, is higher than the cost of a delayed response.
def process_ticket(ticket_body: str):
first = classify_ticket(ticket_body)
final = verify_ticket(ticket_body, first)
if final.priority == "critical" or final.confidence < 7:
return {
"status": "escalate",
"reason": f"Priority {final.priority} with confidence {final.confidence}",
"analysis": final.model_dump(),
}
return {
"status": "reply",
"category": final.category,
"priority": final.priority,
"reply": final.draft_reply,
}
Step 6: Polish the output with a specialist model
For tickets that clear the threshold, I run one last call to qwen-3-32b to turn the draft into a friendly message. Separating classification from generation is a separation of concerns: the first model optimizes for correctness, the second for tone and clarity. Qwen 3 handles multilingual users and agent workflows well, which matters when our user base is global and tickets arrive in mixed languages. Because Oxlo.ai bills per request rather than per token, splitting work across three specialized calls is economically viable. On token-based billing, you would pay for every token in the long system prompt on every single call, which punishes the exact kind of modular, safe architecture we want.
def polish_reply(draft: str, tone: str = "professional") -> str:
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": f"Rewrite the following support draft in a {tone} tone. Do not add new promises or technical steps not in the draft."},
{"role": "user", "content": draft},
],
temperature=0.3,
)
return response.choices[0].message.content
Run it
Here is a realistic security ticket and the full runner. The ticket mentions an unknown IP and unauthorized access, which should trigger the security guardrail immediately. Notice how the pipeline escalates automatically instead of fabricating an answer. The first model catches the security keyword, the verifier confirms critical priority, and the guardrail blocks an unvetted auto-reply from reaching the customer. You can drop in your own ticket text and watch the confidence score change based on the clarity of the request.
if __name__ == "__main__":
ticket = (
"Hi, I noticed an unknown IP logged into our admin dashboard last night. "
"We did not authorize this. Can you check if any data was exported?"
)
result = process_ticket(ticket)
if result["status"] == "escalate":
print("ESCALATED:", result["reason"])
print(json.dumps(result["analysis"], indent=2))
else:
polished = polish_reply(result["reply"])
print(f"Category: {result['category']}")
print(f"Priority: {result['priority']}")
print("---")
print(polished)
Expected output:
ESCALATED: Priority critical with confidence 10
{
"reasoning": "The user mentions an unknown IP accessing the admin dashboard and asks about unauthorized data export. This is a potential security breach.",
"category": "security",
"priority": "critical",
"confidence": 10,
"draft_reply": "Thank you for reporting this immediately. Our security team is investigating."
}
Wrap-up and next steps
This pipeline trades a little latency for a large gain in accuracy. The modular design means you can upgrade individual steps without rewriting the whole agent. The next concrete step is to add few-shot examples to the system prompt for your specific product categories. After that, wire the escalate branch into a Slack webhook so human agents get the full reasoning, not just an alert. If you are currently on a token-based provider, moving this workload to Oxlo.ai will flatten your costs because the per-request price does not scale with prompt length. You can explore the pricing at https://oxlo.ai/pricing and start testing with the free tier that includes 60 requests per day and access to models like deepseek-v3.2.
