Lesson 3: Test and Refine - When AI Gets It Wrong (And What to Do)
Estimated Time: 12 minutes
Learning Objectives:
- Recognize the three most common AI workflow mistakes
- Debug your prompt using the "edit one thing at a time" rule
- Add constraints that fix problems without making your prompt bloated
- Know when to add examples to guide AI's output
Your workflow won't be perfect on day one
Here's what happens after you build your first workflow:
You use it a few times and it's amazing. You're saving time, the outputs are good, and you feel like a genius.
Then AI gives you something that's just... off. Maybe it's too formal. Maybe it misses the point. Maybe it adds weird details you never asked for.
And you think, "Is my workflow broken? Should I start over?"
No. This is normal. This is actually the most important part of the process.
Every AI workflow needs tuning. You build it, you test it with real situations, you find the edge cases where it doesn't work, and you fix those specific problems.
The difference between people who succeed with AI workflows and people who give up is this: successful people debug and refine. They don't expect perfection on version one.
Let's learn how to fix the three most common problems.
Problem 1: AI is too generic (sounds like a robot)
This is the most common issue. You paste in a customer's email, AI writes a response, and it sounds like every corporate customer service email you've ever received.
Example of the problem:
Customer email: "Hey, my order still hasn't shipped and I ordered it 5 days ago. What's going on?"
AI response: "Dear valued customer, we sincerely apologize for the delay. We are looking into your order and will provide an update shortly. Thank you for your patience."
Technically correct. Completely soulless.
Why this happens:
Your constraints aren't specific enough. AI defaults to "professional" which often means "boring and corporate."
How to fix it:
Add tone examples and specific phrases to avoid.
Updated constraint section:
Rules:
- Tone: like you're texting a friend who happens to be a customer, not a corporate FAQ page
- Use contractions (you're, we'll, haven't, not "you are, we will, have not")
- Start with empathy before jumping to the solution
- NEVER use these phrases: "we sincerely apologize," "valued customer," "at this time," "rest assured," "we appreciate your patience"
- DO use phrases like: "Oh no," "Let me check on that," "Here's what I found," "I've got you"
Now try the same customer email again. The AI response becomes:
"Hey [name], oh no! Let me check on that for you. I'm seeing your order in our system from 5 days ago, and it looks like it's still in processing. That's longer than usual. I'm going to flag this for our warehouse team and get you an update by end of day. I'll email you as soon as I know what's happening."
See the difference? Same information, completely different feel. That's the power of specific tone constraints.
Problem 2: AI misses the real question (wrong focus)
Sometimes AI fixates on the wrong part of the email and gives a detailed answer to something the customer didn't actually ask about.
Example of the problem:
Customer email: "I love the shirt I ordered but it's a bit tight. Do you have a size chart anywhere? I want to order another one in the right size."
AI response: "Thank you for your order! We're glad you love the shirt. Our return policy allows exchanges within 30 days. You can initiate a return by visiting our returns portal at..."
The customer didn't ask about returns. They asked for a size chart so they can order correctly next time.
Why this happens:
AI sees keywords ("tight," "order") and jumps to conclusions. It's trying to be helpful but doesn't understand priority.
How to fix it:
Add a step in your task section that forces AI to identify what the customer is actually asking for.
Updated task section:
Task:
1. Read the customer's email and identify their main question or request (state it in one sentence)
2. Answer that specific question directly in your first sentence
3. Then add any relevant additional info they might need
4. Offer to help with anything else
Write a professional but warm email response based on this structure.
Now when you run the same email through:
AI's output:
"Hi [name],
Main question: You need our size chart to order the right size.
Here's our size chart link: [URL]. For that shirt specifically, most people find sizing up one size gives a more relaxed fit, so if you ordered a medium and it's tight, a large would probably be perfect for you.
If you want to exchange the one you have, I'm happy to help with that too - just let me know!
Happy to help,
[Your name]"
Better. It identified the real question, answered it first, then offered additional help without assuming.
Problem 3: AI adds too much or too little detail
Sometimes AI writes a novel when you wanted three sentences. Other times it's so brief it doesn't actually help.
Example of too much:
Customer email: "What's your return policy?"
AI response: "Great question! Our return policy is designed to be as flexible as possible for our customers. We understand that sometimes items don't work out, and we want to make the return process as smooth as possible. Here's how it works: You have 30 days from the date of purchase to return any item. Items must be in original condition with tags attached. Once we receive your return, we'll process the refund within 5-7 business days to your original payment method. If you paid with a credit card, please allow 1-2 billing cycles for the credit to appear on your statement. For international orders, please note that original shipping costs are non-refundable. If you have any other questions about our return policy, feel free to reach out anytime!"
That's 130 words to answer a simple question. The customer is asleep by sentence three.
Example of too little:
Same customer email.
AI response: "30-day returns. Items must be unworn with tags."
Technically answers the question but feels curt. Where do they actually initiate the return? What happens next?
How to fix it:
Add length guidance based on question complexity.
Updated format section:
Format:
- Greeting with their name (1 line)
- Direct answer to their question (1-2 sentences)
- Relevant next steps or additional info (1-2 sentences)
- Offer additional help (1 sentence)
- Friendly closer (1 line)
Total length: 75-125 words (aim for the shorter end unless the question requires more detail)
Now the output is:
"Hi [name],
We have a 30-day return policy on all items. As long as it's unworn with tags still attached, you're good to go.
To start a return, just hit reply and let me know what you'd like to send back, and I'll email you a prepaid shipping label. Refunds usually take 5-7 business days after we receive the item.
Anything else you need?
Happy to help,
[Your name]"
85 words. Answers the question, tells them what to do next, friendly tone. Perfect.
The "edit one thing at a time" rule
When you're debugging your workflow, don't change five things at once. You won't know what actually fixed the problem.
Here's the process:
- Identify the specific problem. Don't just say "this isn't good." Say "the tone is too formal" or "it's not answering the right question."
- Change one constraint. Add a tone rule or a length limit or a structure requirement. Just one.
- Test it with the same email again. See if that one change fixed the issue.
- If it's better, keep the change. If not, try something else.
- Repeat until it works.
This feels slower at first, but it's way faster than randomly editing your prompt and hoping something sticks.
When to add examples to your prompt
Sometimes constraints aren't enough. AI needs to see what "good" looks like.
This is especially true for tone and style. If you want AI to match a very specific voice, show it an example.
Adding a "good example" section:
Example of the tone I want:
Customer question: "Do you ship to Canada?"
Good response:
"Hey [name], yep! We ship to Canada. Shipping usually takes 7-10 business days and costs $15 flat rate. I'll make sure the total is clear at checkout before you finalize anything. Let me know if you have questions!"
Now AI has a reference point. It knows what "warm but professional" actually means in your context.
You don't need examples for every email type. Just add them when AI consistently gets the tone or style wrong despite your constraints.
The Version 2 Challenge
Your workflow is working pretty well now. Maybe 8 out of 10 responses are great with minimal editing.
Here's the challenge: Can you make it 9 out of 10?
Go back to your prompt template and ask yourself:
- Is there a phrase AI keeps using that I always edit out?
- Is there a type of question where AI consistently gets confused?
- Is there a piece of info I always have to add manually?
Fix one of those things. Add a constraint, add an example, adjust the structure. Test it.
That's how you get from "pretty good" to "this is actually saving me massive amounts of time."
What you just learned
- The three most common AI mistakes: too generic, wrong focus, wrong detail level
- How to diagnose and fix each problem with specific prompt changes
- The "edit one thing at a time" rule for debugging
- When to add examples to guide AI's style and tone
Your workflow is no longer just functional. It's refined. It's tuned to your specific needs. It's actually good.
Try It Challenge
Take your workflow template and make it better:
- Run 5 different customer emails through it (or whatever task type you chose)
- For each output, write down one thing you had to edit or wished was different
- Pick the most common issue from those 5
- Edit your prompt to fix that specific problem (add a constraint, adjust the format, include an example)
- Re-run the same 5 emails and see if your change improved things
Document this:
- Version 1 problem: [what kept going wrong]
- Change I made: [constraint or example I added]
- Result: [did it fix it?]
This is how you build a workflow that actually works long-term. You test, you find the weak points, you fix them one at a time.
Quiz
Q1: AI keeps writing responses that are way too long and formal. What's the best way to fix this?
A) Start over with a new prompt from scratch
B) Add specific length limits and tone constraints to your existing prompt
C) Just edit the outputs manually every time
D) Switch to a different AI tool
Answer: B
Don't throw out your whole prompt. Add constraints that address the specific problem. Something like "Keep responses under 100 words" and "Tone: conversational, like texting a friend." Test it and refine from there.
Q2: You changed three things in your prompt at once and now it's worse than before. What should you do?
A) Change three more things to cancel out the first three
B) Undo all three changes and start over, changing one thing at a time
C) Keep the changes and hope it improves over time
D) Give up and go back to writing emails manually
Answer: B
The "edit one thing at a time" rule exists for exactly this reason. When you change multiple things at once, you don't know what helped and what hurt. Undo everything, then make one change, test it, and only add more if that one change worked.
Next lesson: You'll measure your time savings, calculate the real impact of your workflow, and plan your next automation target. This is where you prove to yourself that AI workflows actually work.