FoundationModels tool calling not working (iOS 26, beta 6)

I have a fairly basic prompt I've created that parses a list of locations out of a string. I've then created a tool, which for these locations, finds their latitude/longitude on a map and populates that in the response.

However, I cannot get the language model session to see/use my tool.

I have code like this passing the tool to my prompt:

class Parser {
    func populate(locations: String, latitude: Double, longitude: Double) async {

        let findLatLonTool = FindLatLonTool(latitude: latitude, longitude: longitude)
        let session = LanguageModelSession(tools: [findLatLonTool]) {
            """
                A prompt that populates a model with a list of locations.
            """
            """
                Use the findLatLon tool to populate the latitude and longitude for the name of each location.
            """
        }
        let stream = session.streamResponse(to: "Parse these locations: \(locations)", generating: ParsedLocations.self)
        
        let locationsModel = LocationsModels();
        
        do {
            for try await partialParsedLocations in stream {
                locationsModel.parsedLocations = partialParsedLocations.content
            }
        } catch {
            print("Error parsing")
        }
    }
}

And then the tool that looks something like this:

import Foundation
import FoundationModels
import MapKit

struct FindLatLonTool: Tool {
    typealias Output = GeneratedContent
    
    let name = "findLatLon"
    let description = "Find the latitude / longitude of a location for a place name."
    
    let latitude: Double
    let longitude: Double
    
    @Generable
    struct Arguments {
        @Guide(description: "This is the location name to look up.")
        let locationName: String
    }
    
    func call(arguments: Arguments) async throws -> GeneratedContent {
        let request = MKLocalSearch.Request()
        request.naturalLanguageQuery = arguments.locationName
        request.region = MKCoordinateRegion(
            center: CLLocationCoordinate2D(latitude: latitude, longitude: longitude),
            latitudinalMeters: 1_000_000,
            longitudinalMeters: 1_000_000
        )
        
        let search = MKLocalSearch(request: request)
        let coordinate = try await search.start().mapItems.first?.location.coordinate
        
        if let coordinate = coordinate {
            return GeneratedContent(
                LatLonModel(latitude: coordinate.latitude, longitude: coordinate.longitude)
            )
        }
        
        return GeneratedContent("Location was not found - no latitude / longitude is available.")
    }
}

But trying a bunch of different prompts has not triggered the tool - instead, what appear to be totally random locations are filled in my resulting model and at no point does a breakpoint hit my tool code.

Has anybody successfully gotten a tool to be called?

Answered by Apple Designer in 853553022

Howdy! Looking at this code, I suspect your instructions are the problem rather than the tool itself.

Right now your instructions are:

A prompt that populates a model with a list of locations. Use the findLatLon tool to populate the latitude and longitude for the name of each location.

First I would try phrasing more like:

For each request from the user, get the place names the user mentions and retrieve the coordinates using the finLatLon tool.

Next, I would try cleaning up the prompt by removing the phrase "Parse these locations". I would try with no phrase at all or potentially:

I would like to find: \(locations)

Next, polish some of the tool phrasing for clarity. Currently the tool is named "findLatLon" but "lat" and "lon" could potentially confuse the model. I'd change the tool name to "getCoordinates".

Finally, I would change the tool description to better match your input arguments. Currently you have:

Find the latitude / longitude of a location for a place name

I'd change this to:

Get the coordinates based on a location name

since your input arg is called locationName. I hope this helps!

Howdy! Looking at this code, I suspect your instructions are the problem rather than the tool itself.

Right now your instructions are:

A prompt that populates a model with a list of locations. Use the findLatLon tool to populate the latitude and longitude for the name of each location.

First I would try phrasing more like:

For each request from the user, get the place names the user mentions and retrieve the coordinates using the finLatLon tool.

Next, I would try cleaning up the prompt by removing the phrase "Parse these locations". I would try with no phrase at all or potentially:

I would like to find: \(locations)

Next, polish some of the tool phrasing for clarity. Currently the tool is named "findLatLon" but "lat" and "lon" could potentially confuse the model. I'd change the tool name to "getCoordinates".

Finally, I would change the tool description to better match your input arguments. Currently you have:

Find the latitude / longitude of a location for a place name

I'd change this to:

Get the coordinates based on a location name

since your input arg is called locationName. I hope this helps!

I am also experiencing issues with tool calling. Not sure what is the issue yet, but what has been answered to you is checked on my code. It used to work perfectly before the change to the new tool output format.

FoundationModels tool calling not working (iOS 26, beta 6)
 
 
Q