Skip to main content

Debugging Techniques

Master systematic debugging approaches using execution logs and AI troubleshooting tools.

Overview​

When agents don't behave as expected, systematic debugging helps you identify and fix issues quickly. ExecAssist provides detailed execution logs and AI-powered analysis to streamline troubleshooting.

The Debugging Process​

Step 1: Identify the Problem​

Start by clearly defining what's wrong:

  • What did you expect to happen?
  • What actually happened?
  • When did it start occurring?
  • Does it happen consistently or intermittently?

Step 2: Gather Evidence​

Collect relevant information:

  1. Execution ID - Specific execution that shows the issue
  2. Time range - When did the problem occur?
  3. Affected agent - Which agent is misbehaving?
  4. Email details - What triggered the execution?

Step 3: Analyze a Run​

Examine the execution log thoroughly:

  • Input (the email)
  • Agent reasoning
  • Tool calls and results
  • Final output or error

Step 4: Form a Hypothesis​

Based on evidence, propose what's causing the issue:

  • Instruction problem?
  • Missing tool?
  • Wrong data?
  • External service issue?

Step 5: Test and Verify​

  • Make a targeted change
  • Test with similar input
  • Verify the fix
  • Monitor for recurrence

Using the Run Inspector​

Opening a Run​

  1. Go to Governance > Activity History > Agent Runs
  2. Find the problematic run
  3. Select it to open the run view

Understanding the Run View​

  • Status - The run's current status
  • Duration - How long the run took
  • Counts - Model calls, tool calls and failed calls
  • Cost - Calculated cost for this run

Trace​

Follow the run's steps in order. Expand phases and select a step to inspect what happened without losing your place in the run.

Step Inspector​

The tabs depend on the selected step:

  • Prompt and Response - Model input and output
  • Tokens - Token usage for a model call
  • Input and Output - Tool parameters and results
  • Raw log - The log entries behind the step

Key Things to Look For​

In Agent Reasoning​

  • Is the agent understanding the email correctly?
  • Is it making appropriate decisions?
  • Does the reasoning align with instructions?

In Tool Calls​

  • Are the right tools being called?
  • Are parameters correct?
  • Are results being used appropriately?

In the Response​

  • Does the output match expectations?
  • Are there obvious errors?
  • Is the format correct?

AI-Powered Troubleshooting​

Using the Summary and Issues​

The run view combines an AI summary with issues and likely fixes:

  1. Open a run with issues
  2. Review the summary card. When the run has finished and logs are available, select Summarize to generate a saved summary or Regenerate to replace it
  3. Review the issues and likely fix
  4. Select an issue's step link to inspect the related step

What to Review​

  • Summary - An overview and key actions from the run
  • Issues - Failed or held steps that need attention
  • Likely fix - Guidance for recognized problems
  • Step details - Inputs, outputs and raw log entries that support your diagnosis

When to Use AI Troubleshooting​

  • Execution failed with an error
  • Agent made obviously wrong decisions
  • Response quality is poor
  • You're unsure what's wrong

Common Debugging Scenarios​

Scenario: Agent Ignores Important Information​

Symptoms: Agent response doesn't reference key email content

Debugging steps:

  1. Check if the information appears in the prompt
  2. Review agent reasoning for any mention
  3. Check if instructions tell agent to look for this info
  4. Test with explicit instructions about this information

Likely fixes:

  • Add explicit instructions about what to look for
  • Provide examples of desired behavior
  • Highlight important email elements in instructions

Scenario: Wrong Tool Called​

Symptoms: Agent calls search when it should reply, or vice versa

Debugging steps:

  1. Review which tool was called and why
  2. Check agent reasoning for decision-making
  3. Compare to instructions about tool usage
  4. Test instructions about when to use each tool

Likely fixes:

  • Clarify tool usage conditions in instructions
  • Add decision criteria for tool selection
  • Provide examples of correct tool usage

Scenario: Tool Returns Error​

Symptoms: Tool call fails with an error message

Debugging steps:

  1. Check the error message in tool result
  2. Review parameters passed to the tool
  3. Test the tool manually if possible
  4. Check external service status

Likely fixes:

  • Fix tool configuration
  • Adjust how agent formats parameters
  • Handle tool errors in instructions

Scenario: Response Too Long/Short​

Symptoms: Responses are consistently wrong length

Debugging steps:

  1. Review response length in execution
  2. Check token counts
  3. Review length guidance in instructions
  4. Check max_tokens setting

Likely fixes:

  • Add explicit length guidance in instructions
  • Adjust max_tokens parameter
  • Provide example responses of correct length

Scenario: Inconsistent Behavior​

Symptoms: Same input produces different outputs

Debugging steps:

  1. Compare multiple executions with similar input
  2. Check temperature setting
  3. Review if context differs between executions
  4. Look for external factors (time, VIP status, etc.)

Likely fixes:

  • Lower temperature for more consistency
  • Clarify ambiguous instructions
  • Add more specific decision criteria

Debugging Tools and Techniques​

Comparing Executions​

  1. Open two execution tabs side by side
  2. Compare inputs - Are they really the same?
  3. Compare reasoning - Where do decisions diverge?
  4. Compare outputs - What's different?

Testing Instruction Changes​

  1. Edit agent instructions
  2. Save changes
  3. Send test email
  4. Compare new execution to old
  5. Iterate until fixed

Using Filters Effectively​

In Agent Runs, use the controls to find patterns:

  • Failed or Attention - Find failures or runs that need attention
  • Source and Agent - Focus on one agent
  • Time range - Narrow to the problem timeframe
  • Search - Find a subject, person, tool or run ID

Examining Raw Data​

For deep debugging:

  1. View the raw request JSON
  2. Check exact prompt format
  3. Review raw response
  4. Look for unexpected content

Debugging Checklist​

Use this checklist when debugging:

  • Identified specific execution with issue
  • Reviewed execution timeline
  • Checked email input for issues
  • Reviewed agent reasoning
  • Verified correct tools were called
  • Checked tool results for errors
  • Compared to expected behavior
  • Identified likely root cause
  • Made targeted fix
  • Verified fix with test execution

Prevention: Writing Debuggable Instructions​

Make future debugging easier:

Be Explicit About Decisions​

When deciding whether to escalate:
- IF the customer mentions "lawyer" or "legal", escalate immediately
- IF the issue involves money over $500, escalate
- IF you cannot find an answer in 2 searches, escalate
- OTHERWISE, continue handling normally

Include Reasoning Requests​

Before responding, briefly explain your reasoning:
1. What is the customer asking for?
2. What information do I need?
3. Can I resolve this, or should I escalate?
Then proceed with your response.

Add Logging Points​

When searching the knowledge base:
- Note what you're searching for and why
- If no results, try an alternative query
- If still no results, note this and proceed with available info

When to Escalate​

Contact support when:

  • Same error persists across many executions
  • Issue started after a system update
  • Error messages reference system components
  • You've tried all relevant debugging steps
  • Issue affects multiple agents or mailboxes

Provide:

  • Execution IDs showing the issue
  • Steps you've tried
  • Time when issue started
  • Any recent changes made