I'm trying to use Foundation Models to identify the things in an image. That part is easy and is working well.
I'd like to also know where in the image the things are. This is where I'm hitting a wall.
For example, if there's an image of a horse and a cow, I'd like to be told (even approximate) coordinates of where in the image the horse is and where the cow is. Bounding boxes are fine for my needs. (Any coordinate system will work because it's easy enough to convert from one to another)
The LanguageModelSession consistently lists the items in the image and gives me bounding boxes for their location that are reasonable approximations of where the images are in relation to one another, but it will (usually, not always) completely fail at explaining where the objects are in relation to the image as a whole, which is what I need.
What's more, the failures are not consistent. Sometimes it will tell me that all the images are in the top half of the image. Other times, it will blow up the location of one or more objects in the image to multiples of their actual size.
I've tried asking the LanguageModelSession to output the locations in various coordinate systems:
raw pixel numbers
normalized position (0 ... 1)
integer percent position (0% ... 100%)
a few different attempts at "soft location" systems where I just ask the LLM to tell me if the objects are in the top left corner or in the center for instance
Of these, the "soft location" gives more consistent answers, but nothing that is complete enough to be usable. Asking for raw pixels gives answers that are ALMOST usable, but the position rectangles it gives are often off by one or two times the width or height of the object or suffer from the issue of "bunching" all the rectangles into the top of the image.
I believe that part of the problem I'm having is that FoundationModels must downsample the image before processing. It appears that it's downsampling to 896px for the longest dimension of the image. Even accounting for this, though, I get strange output.
Yes, I have considered using VisionKit's GenerateObjectnessBasedSaliencyImageRequest. It works well for another part of my project, but it doesn't fit exactly the particular need that I have here. It gives me locations of objects but not what they are. FoundationModels gives me what objects are in the image but not their locations.
It may be that FoundationModels just isn't going to give me an accurate enough location for the objects in the image. It's a LLM, not a ML model, after all. If that's the case, I'd appreciate if someone would verify that so I can stop barking up this tree. It just seems like it should be possible, and I keep getting results that are almost accurate enough to be useful to me.
Any help at all would be appreciated. Below are the instructions and prompt I'm using.
let session = LanguageModelSession(
instructions: """
You describe images to help another AI model identify and label distinct objects.
Identify the distinct foreground subjects — objects, animals, people, or things
that stand out as individual items someone would point to and name.
Be specific (e.g. "a black and white cow", "a red coffee mug", "a wooden chair").
For each subject, provide a tight bounding box as pixel coordinates:
- topLeft: upper-left corner of the box (x from left edge, y from top edge)
- bottomRight: lower-right corner (x and y must be larger than topLeft's)
- (0, 0) is the top-left pixel; x increases rightward, y increases downward
- the exact pixel dimensions of each image are stated in the prompt you receive
Also note background objects — items visible in the scene but not the main focus.
Describe the setting — the background environment (surface, room, landscape, or space).
Do not merge subjects and setting. A cow standing in a field has the cow as a subject
and the field as the setting — not both as subjects.
"""
)
let prompt = Prompt {
"Describe this \(imageWidth)×\(imageHeight) image. Bounding box coordinates are in pixels: (0,0) is top-left, (\(imageWidth),\(imageHeight)) is bottom-right."
Attachment(modelImage.cgImage, orientation: modelImage.orientation)
}
