Skip to main content

Debugging Techniques

Master systematic debugging approaches using execution logs and AI troubleshooting tools.

Overview​

When agents don't behave as expected, systematic debugging helps you identify and fix issues quickly. ExecAssist provides detailed execution logs and AI-powered analysis to streamline troubleshooting.

The Debugging Process​

Step 1: Identify the Problem​

Start by clearly defining what's wrong:

  • What did you expect to happen?
  • What actually happened?
  • When did it start occurring?
  • Does it happen consistently or intermittently?

Step 2: Gather Evidence​

Collect relevant information:

  1. Execution ID - Specific execution that shows the issue
  2. Time range - When did the problem occur?
  3. Affected agent - Which agent is misbehaving?
  4. Email details - What triggered the execution?

Step 3: Analyze Execution Details​

Examine the execution log thoroughly:

  • Input (the email)
  • Agent reasoning
  • Tool calls and results
  • Final output or error

Step 4: Form a Hypothesis​

Based on evidence, propose what's causing the issue:

  • Instruction problem?
  • Missing tool?
  • Wrong data?
  • External service issue?

Step 5: Test and Verify​

  • Make a targeted change
  • Test with similar input
  • Verify the fix
  • Monitor for recurrence

Using Execution Details​

  1. Go to Operations > Activity > Agent Executions
  2. Find the problematic execution
  3. Click to open details

Understanding the Execution View​

Header Section​

  • Status - Success, Failed, or In Progress
  • Duration - How long the execution took
  • Tokens - Input and output token counts
  • Cost - Calculated cost for this execution

Timeline View​

Shows the sequence of events:

  1. Email Received - Original email content
  2. Agent Reasoning - LLM's thinking process
  3. Tool Calls - Each tool invocation with parameters and results
  4. Response Generated - Final output

Request/Response Details​

  • Full prompt - Complete input sent to LLM
  • Raw response - Exact LLM output
  • Error details - If execution failed

Key Things to Look For​

In Agent Reasoning​

  • Is the agent understanding the email correctly?
  • Is it making appropriate decisions?
  • Does the reasoning align with instructions?

In Tool Calls​

  • Are the right tools being called?
  • Are parameters correct?
  • Are results being used appropriately?

In the Response​

  • Does the output match expectations?
  • Are there obvious errors?
  • Is the format correct?

AI-Powered Troubleshooting​

Using the Troubleshoot Feature​

ExecAssist includes AI-powered analysis for failed or problematic executions:

  1. Open an execution with issues
  2. Click Troubleshoot with AI
  3. Review the analysis
  4. Follow suggested fixes

What AI Analysis Provides​

  • Root cause identification - Why the execution failed
  • Instruction suggestions - How to improve agent instructions
  • Tool recommendations - Which tools might help
  • Similar issues - Patterns from other executions

When to Use AI Troubleshooting​

  • Execution failed with an error
  • Agent made obviously wrong decisions
  • Response quality is poor
  • You're unsure what's wrong

Common Debugging Scenarios​

Scenario: Agent Ignores Important Information​

Symptoms: Agent response doesn't reference key email content

Debugging steps:

  1. Check if the information appears in the prompt
  2. Review agent reasoning for any mention
  3. Check if instructions tell agent to look for this info
  4. Test with explicit instructions about this information

Likely fixes:

  • Add explicit instructions about what to look for
  • Provide examples of desired behavior
  • Highlight important email elements in instructions

Scenario: Wrong Tool Called​

Symptoms: Agent calls search when it should reply, or vice versa

Debugging steps:

  1. Review which tool was called and why
  2. Check agent reasoning for decision-making
  3. Compare to instructions about tool usage
  4. Test instructions about when to use each tool

Likely fixes:

  • Clarify tool usage conditions in instructions
  • Add decision criteria for tool selection
  • Provide examples of correct tool usage

Scenario: Tool Returns Error​

Symptoms: Tool call fails with an error message

Debugging steps:

  1. Check the error message in tool result
  2. Review parameters passed to the tool
  3. Test the tool manually if possible
  4. Check external service status

Likely fixes:

  • Fix tool configuration
  • Adjust how agent formats parameters
  • Handle tool errors in instructions

Scenario: Response Too Long/Short​

Symptoms: Responses are consistently wrong length

Debugging steps:

  1. Review response length in execution
  2. Check token counts
  3. Review length guidance in instructions
  4. Check max_tokens setting

Likely fixes:

  • Add explicit length guidance in instructions
  • Adjust max_tokens parameter
  • Provide example responses of correct length

Scenario: Inconsistent Behavior​

Symptoms: Same input produces different outputs

Debugging steps:

  1. Compare multiple executions with similar input
  2. Check temperature setting
  3. Review if context differs between executions
  4. Look for external factors (time, VIP status, etc.)

Likely fixes:

  • Lower temperature for more consistency
  • Clarify ambiguous instructions
  • Add more specific decision criteria

Debugging Tools and Techniques​

Comparing Executions​

  1. Open two execution tabs side by side
  2. Compare inputs - Are they really the same?
  3. Compare reasoning - Where do decisions diverge?
  4. Compare outputs - What's different?

Testing Instruction Changes​

  1. Edit agent instructions
  2. Save changes
  3. Send test email
  4. Compare new execution to old
  5. Iterate until fixed

Using Filters Effectively​

In the Executions list, use filters to find patterns:

  • By status - Find all failures
  • By agent - Focus on one agent
  • By date - Narrow to problem timeframe
  • By duration - Find unusually slow executions

Examining Raw Data​

For deep debugging:

  1. View the raw request JSON
  2. Check exact prompt format
  3. Review raw response
  4. Look for unexpected content

Debugging Checklist​

Use this checklist when debugging:

  • Identified specific execution with issue
  • Reviewed execution timeline
  • Checked email input for issues
  • Reviewed agent reasoning
  • Verified correct tools were called
  • Checked tool results for errors
  • Compared to expected behavior
  • Identified likely root cause
  • Made targeted fix
  • Verified fix with test execution

Prevention: Writing Debuggable Instructions​

Make future debugging easier:

Be Explicit About Decisions​

When deciding whether to escalate:
- IF the customer mentions "lawyer" or "legal", escalate immediately
- IF the issue involves money over $500, escalate
- IF you cannot find an answer in 2 searches, escalate
- OTHERWISE, continue handling normally

Include Reasoning Requests​

Before responding, briefly explain your reasoning:
1. What is the customer asking for?
2. What information do I need?
3. Can I resolve this, or should I escalate?
Then proceed with your response.

Add Logging Points​

When searching the knowledge base:
- Note what you're searching for and why
- If no results, try an alternative query
- If still no results, note this and proceed with available info

When to Escalate​

Contact support when:

  • Same error persists across many executions
  • Issue started after a system update
  • Error messages reference system components
  • You've tried all relevant debugging steps
  • Issue affects multiple agents or mailboxes

Provide:

  • Execution IDs showing the issue
  • Steps you've tried
  • Time when issue started
  • Any recent changes made