Apply Prompt Engineering: What Works

Overview

Large Language Models (LLMs) are general-purpose reasoning engines, but their output quality depends heavily on how you ask the question. Prompt engineering is the practice of structuring inputs to guide the model toward accurate, relevant, and well-formatted responses. Rewording a single sentence can turn a vague paragraph into a precise JSON object.

I’ve spent more time than I’d like to admit debugging “the model is broken” issues that turned out to be a missing example or an ambiguous instruction in the prompt. The techniques below are the ones I reach for first: assigning a role, providing few-shot examples, requesting chain-of-thought reasoning, and constraining output format. They work across GPT-4, Claude, Gemini, and open-source models like Llama.

When to Use

Use this recipe when:

• Building applications that call LLM APIs for classification, extraction, or generation
• Debugging inconsistent or hallucinated model outputs
• Designing chatbots, copilots, or AI-powered assistants
• Implementing automated content moderation, summarization, or translation pipelines
• Evaluating prompt versions with A/B testing frameworks

Solution

Role Assignment (System Prompt)

import openai

response = openai.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are a senior Python code reviewer. Be concise, focus on security and performance issues."},
        {"role": "user", "content": "Review this function: def login(email, password): ..."}
    ]
)

Few-Shot Examples

response = openai.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "Classify user intent into: SEARCH, SUPPORT, BILLING, or OTHER."},
        {"role": "user", "content": "How do I reset my password?"},
        {"role": "assistant", "content": "SUPPORT"},
        {"role": "user", "content": "Find me red running shoes under $100"},
        {"role": "assistant", "content": "SEARCH"},
        {"role": "user", "content": "I was charged twice last month"},
        {"role": "assistant", "content": "BILLING"},
        {"role": "user", "content": user_input},
    ]
)

Chain-of-Thought Reasoning

response = openai.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "Solve math problems step by step. Show your reasoning, then give the final answer on the last line prefixed with ANSWER:"},
        {"role": "user", "content": "If a train travels 120 km in 2 hours, how far will it travel in 5 hours at the same speed?"}
    ]
)

Structured JSON Output

response = openai.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "Extract entities from the text. Respond ONLY with valid JSON matching this schema: {\"person\": string, \"organization\": string, \"location\": string}"},
        {"role": "user", "content": "Elon Musk announced that Tesla will build a new factory in Mexico."}
    ],
    response_format={"type": "json_object"}
)

Function Calling (Tool Use)

response = openai.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "user", "content": "What is the weather in Tokyo right now?"}
    ],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"}
                },
                "required": ["city"]
            }
        }
    }],
    tool_choice="auto"
)

# The model returns a tool_call; you execute it and feed results back
tool_call = response.choices[0].message.tool_calls[0]
# Execute get_weather("Tokyo") in your code, then send the result back

Go (Using langchaingo)

package main

import (
    "context"
    "fmt"
    "github.com/tmc/langchaingo/llms"
    "github.com/tmc/langchaingo/llms/openai"
)

func main() {
    llm, err := openai.New()
    if err != nil {
        panic(err)
    }

    ctx := context.Background()
    resp, err := llms.GenerateFromSinglePrompt(ctx, llm,
        "You are a code reviewer. Review this function for security issues.\n"+
        "func login(email, password string) error { ... }",
        llms.WithTemperature(0),
    )
    if err != nil {
        panic(err)
    }
    fmt.Println(resp)
}

Prompt Chaining (Multi-Step Pipeline)

# Step 1: Extract key topics from a document
extract_response = openai.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "Extract 3-5 key topics from the text as a JSON array of strings."},
        {"role": "user", "content": long_document_text}
    ],
    response_format={"type": "json_object"}
)
topics = extract_response.choices[0].message.content

# Step 2: Generate a summary for each topic
summary_response = openai.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "Write a 2-sentence summary for each topic provided. Return as JSON: {topic: summary}."},
        {"role": "user", "content": topics}
    ],
    response_format={"type": "json_object"}
)

Explanation

• Role assignment: LLMs adapt tone, depth, and format based on the persona you assign. Ask the same question as a “legal expert” and as a “friendly tutor” and you get different answers.
• Few-shot learning: Providing input/output examples in the context teaches the model your expected format without fine-tuning. Three to five examples usually suffice.
• Chain-of-thought: Explicitly asking the model to reason step-by-step dramatically improves accuracy on complex tasks (math, logic, multi-step planning). Debugging gets easier too — you can see the exact step where the reasoning derailed.
• Structured output: forcing responses into JSON, XML, or a fixed format kills parsing errors and keeps downstream processing reliable.
• Function calling: Instead of parsing free-text responses to determine actions, the model returns structured tool calls with typed parameters. Your code runs the function and feeds the result back — that loop is what makes agentic workflows possible.
• Prompt chaining: Breaking complex tasks into sequential steps (extract → summarize → format) produces better results than a single mega-prompt. Every step gets only the context it needs, and you can test each one on its own.

Variants

Technique	Use Case	Cost Impact
Zero-shot	Simple classification, Q&A	Low tokens
Few-shot	Format-specific extraction	Medium tokens
Chain-of-thought	Complex reasoning, math	Higher tokens
Function calling	Tool use, API integration	Medium tokens
Prompt chaining	Multi-step pipelines	Higher tokens (one call per step)
Self-consistency	Math, logic (sample N times, majority vote)	N× cost
ReAct (Reason+Act)	Agentic workflows with tools	High tokens

What Works

• Be specific and explicit: vague prompts produce vague answers. Instead of “summarize this,” try “summarize in 3 bullet points focusing on financial impact.”
• Use delimiters for long inputs: wrap user content in XML tags (<article>...</article>) or triple backticks. Skip them and the model can’t reliably tell your instructions apart from the data.
• Set temperature per task: I use 0 for deterministic work (classification, extraction) and 0.7+ for anything creative.
• Validate and sanitize outputs: LLMs can hallucinate, produce invalid JSON, or ignore instructions. Always parse defensively and have fallback logic.
• Version and track prompts: store prompts in version control. One swapped word can tank output quality, and you want a way back.
• Test across different models: a prompt that works on GPT-4 may fail on Llama or Claude.
• Use system prompts for fixed instructions: put role, format, and constraint instructions in the system message rather than the user message. That keeps token usage down on later turns and stops the instructions drifting mid-conversation.
• Build a prompt evaluation suite: a script that runs your prompt against a 20-50 input test set and reports pass/fail. Run it before and after each prompt change; it catches regressions before production does. A runnable version lives in the companion repo.
• Use separate prompts for separate tasks: a single prompt that tries to classify, summarize, and extract entities will do all three poorly. Split into separate API calls with focused prompts.
• Cache responses for deterministic prompts: if the same prompt + input always produces the same output (temperature=0), cache the response to eliminate redundant API calls.
• Monitor prompt drift: watch output quality metrics over time. When quality drops and you changed nothing, the provider probably shipped a silent model update.
• Log full request/response pairs: store the complete prompt, model, parameters, and response for each API call — essential for debugging, auditing, and improving prompts over time.

Common Mistakes

• Overloading context: sending 50 examples wastes tokens and can confuse the model. Keep only the examples that matter.
• Trusting outputs without validation: LLMs will confidently return wrong answers. Verify facts yourself — doubly so in medicine or finance, where a confident wrong answer costs real money.
• Ignoring token limits: a prompt that eats 10,000 tokens barely leaves room for the model to answer.
• Not handling refusals: some queries trigger safety filters. Your application should gracefully handle refusals and partial responses.
• Ambiguous instructions: “make it better” or “fix this” gives the model nothing to act on. Spell it out — shorter? more formal? compliant with a style guide?
• Inconsistent few-shot format: examples with mixed formatting teach the model noise, not the pattern.
• Not setting max_tokens: leave it uncapped and the model happily writes twice what you need — you pay for those tokens and wait for them. Cap it at your expected length plus headroom.
• Prompt injection from user input: if your prompt includes user-generated text, a malicious user can inject instructions like “ignore all previous instructions.” Sanitize and delimit user content.
• Not testing edge cases: prompts that work on typical inputs may fail on empty strings, very long text, special characters, or non-English input.
• Not using stop sequences: without them the model rambles past the answer. Set stop sequences (e.g., "\n\n", "###") to cleanly terminate responses.
• Ignoring the cost of few-shot examples: each example adds tokens to every API call — for high-volume applications, 5 examples per call adds up fast.
• One temperature for everything: 0 for classification, 0.7+ for creative writing, 0.2 for code — pick per task, not per project.
• Not handling model deprecation: providers deprecate older models and your prompts may behave differently on the replacement. Test prompts against new models before switching.
• Overcomplicating system prompts: a 500-word system prompt with 20 rules is harder to follow than a 100-word prompt with 5 key rules.

Troubleshooting

• Model outputs are inconsistent: set temperature=0 for deterministic tasks, use seed where supported, and version the prompt.
• Prompt injection leaks context: wall off user input from system instructions with delimiters and treat it as data, never as commands.
• High token costs: cache responses for identical inputs, shorten system prompts, and use smaller models (GPT-4o-mini) for simple tasks.
• JSON output is malformed: turn on response_format={"type": "json_object"} and still validate the schema server-side; valid JSON can hold semantically wrong values.
• Few-shot examples ignored: check that your examples use consistent formatting; inconsistent patterns confuse the model more than they help.