How does OrchestrationHandoffs manage threads and chat history?

[rmbayer]

Python==3.10
Semantic-kernel==1.37.0
Language Model = gpt-4o

Details of Agents at the end in Project Details

Description:
I'm having issues getting Orchestrations to perform as I would expect. Specifically, I am now very involved with the OrchestrationHandoffs() utilizing a ChatCompletionAgent().

I have agents that I have built from ChatCompletionAgent() using an AzureChatCompletion() service. Alone these agents work well. They each have their own Kernel and ChatHistoryAgentThread.
Now I am trying to have an orchestrator utilize handoffs and am getting hallucinations and less desirable results.

Example 1:
If I say "show me 20 records for New York", the SQL agent alone will use it's system instructions to lookup the code for New York and create and run a SQL query, then explain a summary of the data to the user.
I can then keep the conversation going and say, I also only want to look at Brooklyn, and the agent will keep the same "SELECT TOP 20... Where State=" and also add the appropriate "AND CITY="

When I now wrap this in a OrchestrationHandoffs() agent I see multiple issues:

• Orchestration agents use a InProcessRuntime and do not have a chat history, so follow-up questions are treated as completely new questions.
• The Orchestration agent does not use the response from the SQL Agent, instead all the user sees is "Task is completed with summary: No handoff agent name provided and no human response function set. Ending task."
• Now the underlying agents are making mistakes and hallucinating column names that don't exist and are trying to pass arguments that don't exist to functions

Example 2:
If I say "Show me 20 records for new york and create a graph", it will call the SQL agent to pull the data and produce a graph, but then it just says "here is the graph".
Calling the SQLAgent on it's own would say "I pulled the data, I have records for ... and they look like ..." and then if those were saved to a .csv and I called the GraphAgent on it's own it would say "I created a graph visualizing xyz"

Questions I have:

• How is a conversation thread managed with a OrchestrationHandoffs() agent? Can I access a ChatHistory() of this agent?
• Does the orchestration directly just push the prompt to the other agents or does it reformat it?
  • The only thing I can see in logs is INFO:semantic_kernel.kernel:Calling Handoff-transfer_to_SQLAgent function with args: {}
• Is the orchestration directly calling the underlying tools? or is it calling the agent.invoke()?
• How can I manipulate the Chat conversation to show the final results of each agent before a handoff?

Project Details:

4 Agent system

• Orchestrator
• Sql agent
• Graph agent
• Analyzer agent

SQL Agent: Has database Schema, join relationships, examples, and a set of tools. It's role is to pull the necessary data for other operations, or just answer basic questions.
Tools:

• generate_and_execute_sql
• get_available_location
• load_more_records
• get_sample_data
  Alone, this agent works great when just using .invoke(). It gets queries correct, explains the data it retrieved, and stores a pd.dataframe in memory.

Graph Agent: Retrieves the records pulled by the SQLAgent and can create graphs based on examples it has in it's prompt
Tools:

• create_matplotlib_graph
  Again, this agent alone works great by loading a .csv or a pandas dataframe

Analyzer Agent: Has specific pd.dataframe operations that it performs based on the data available to understand what data there is.

• analyze_dataframe
  Again, this agent alone works great by loading a .csv or a pandas dataframe

[oiluis]

I know I am late to the party, but any success with it ?

[KeepALifeUS]

The core issue you're hitting is that the orchestration layer abstracts away state management, making it opaque.

Alternative: External State Management

Instead of relying on the framework's internal thread handling, manage state explicitly:

state = {
    "conversation_id": "abc123",
    "chat_history": [],
    "agent_outputs": {},
    "handoff_context": {}
}

# Before each agent call
def prepare_agent_context(agent_name, state):
    return {
        "history": state["chat_history"],
        "previous_agent": state.get("last_agent"),
        "context": state["handoff_context"]
    }

# After each agent completes
def record_agent_output(agent_name, output, state):
    state["agent_outputs"][agent_name] = output
    state["chat_history"].append({"role": "assistant", "agent": agent_name, "content": output})
    state["last_agent"] = agent_name

Why This Solves Your Issues

1. Chat history persists — It's in your state, not the framework's internal runtime

2. Follow-ups work — Read history from state, inject into context

3. Responses are visible — You control what gets recorded and displayed

4. Hallucinations reduce — Agents get explicit context, not reconstructed prompts

The Pattern

This is stigmergy — agents coordinate through shared state rather than framework-managed threads. Benefits:

• Full visibility into what each agent sees
• Easy to debug (just inspect state)
• Framework-agnostic (works with any orchestrator)

Working example: https://github.com/KeepALifeUS/autonomous-agents