In my previous post, we talked about when and where to use agents in an LLM application. We ended the post by saying that if we need an answer for “why something happened?”, then an agentic framework is necessary.
Today, we will discuss on how structured outputs from an LLM can help in invoking the next course of action.
A structured machine readable output is essential to build a real AI application. A straight forward and free-text answer from an LLM is not useful when you want to trigger an action based on the response. We can obtain the structured output in two ways, 1. JSON Mode 2. Function Calling
Press enter or click to view image in full size
JSON Mode:
This is the simpler approach for getting machine readable data. Let’s say your automated dashboard and the narratives obtained using deterministic code says, “Sales decreased by 10%. South Region sales decreased by 18%. Product A sales decreased by 25%.”
To know “Why the sales decreased” and “what should you do next” you can pass this text to LLM and ask for a structured output. Input: — — — - What should I investigate next? Return JSON: { “next_action”: “”, “reason”: “” }
Output: — — — — - { “next_action”: “investigate_product”, “reason”: “Product A had the largest decline” }
Your code can then do: if(response.next_action === “investigate_product”) { investigateProduct(); }
If you wish to investigate further then, you can ask the LLM to “investigate region” and continue the process.
Function Calling:
Function Calling is a mechanism that allows the LLM to choose and invoke business logic functions. This mechanism is widely used for agent frameworks where the workflow is usually not known and must be discovered dynamically. For the question, “why did sales drop?”, LLM decides what should be investigated. Region? Product? Customer? Channel?
The LLM decides: { “function”: “calculateRegionalInsights”, “arguments”: {} } Your app runs: calculateRegionalInsights() Results come back: South Region dropped 18%
Now the LLM decides: { “function”: “calculateProductInsights”, “arguments”: { “region”: “South” } } Your app runs: calculateProductInsights(“South”)
To Conclude, JSON Mode is enough if you are comfortable taking the LLM’s decision and writing the orchestration code yourself. Function Calling becomes useful when you have many tools and want the framework to automatically route the LLM’s decisions to the appropriate functions.
Note: I have used JSON mode in my deterministic logic using JS. Function calling was best to generate narratives and recommendations.
Let me know in the comments which method has been most useful to you in your application.