Skip to content

Throw when no requested model preference is available #245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alewolf wants to merge 3 commits into
base: trunk
Choose a base branch
from
+173 −13
Open

Throw when no requested model preference is available #245

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4ce39ee
Throw when no requested model preference is available.
alewolf Jun 4, 2026
5447cd4
Use polyfilled str_starts_with() to match house style.
alewolf Jun 4, 2026
24e2707
Address review: harden preference parsing, clarify message, cover tup…
alewolf Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 69 additions & 10 deletions src/Builders/PromptBuilder.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,11 @@ public function usingModel(ModelInterface $model): self
/**
* Sets preferred models to evaluate in order.
*
* The preferences are evaluated in order, and the first one that is available for the prompt's
* requirements is used. If none of the provided preferences are available, generation throws an
* InvalidArgumentException rather than silently falling back to an arbitrary (potentially far more
* expensive) model. To allow automatic discovery instead, omit this method entirely.
*
* @since 0.2.0
*
* @param string|ModelInterface|array{0:string,1:string} ...$preferredModels The preferred models as model IDs,
Expand Down Expand Up @@ -1320,7 +1325,8 @@ protected function appendPartToMessages(MessagePart $part): void
*
* @param CapabilityEnum $capability The capability the model will be using.
* @return ModelInterface The model to use.
* @throws InvalidArgumentException If no suitable model is found or set model doesn't meet requirements.
* @throws InvalidArgumentException If no suitable model is found, the set model doesn't meet requirements, or
* explicit model preferences were provided but none are available.
*/
private function getConfiguredModel(CapabilityEnum $capability): ModelInterface
{
Expand Down Expand Up @@ -1363,18 +1369,24 @@ private function getConfiguredModel(CapabilityEnum $capability): ModelInterface
$candidateMap
);

if (!empty($matchingPreferences)) {
// Get the first matching preference key
$firstMatchKey = key($matchingPreferences);
[$providerId, $modelId] = $candidateMap[$firstMatchKey];

$model = $this->registry->getProviderModel($providerId, $modelId, $this->modelConfig);
$this->bindModelRequestOptions($model);
return $model;
if (empty($matchingPreferences)) {
// Explicit model preferences were provided, but none of them are available from the
// resolved provider(s). Silently falling back to an arbitrary discovered model would
// route the request to a different (potentially far more expensive) model than the
// caller asked for, with no signal. Fail loudly instead. See issue #241.
throw new InvalidArgumentException($this->buildUnmatchedPreferenceMessage($capability));
}

// Get the first matching preference key
$firstMatchKey = key($matchingPreferences);
[$providerId, $modelId] = $candidateMap[$firstMatchKey];

$model = $this->registry->getProviderModel($providerId, $modelId, $this->modelConfig);
$this->bindModelRequestOptions($model);
return $model;
}

// No preference matched; fall back to the first candidate discovered.
// No preference specified; fall back to the first candidate discovered.
[$providerId, $modelId] = reset($candidateMap);

$model = $this->registry->getProviderModel($providerId, $modelId, $this->modelConfig);
Expand Down Expand Up @@ -1519,6 +1531,53 @@ private function createModelPreferenceKey(string $modelId): string
return 'model::' . $modelId;
}

/**
* Builds the exception message for when no requested model preference is available.
*
* Reconstructs a human-readable list of the requested preferences from their internal
* keys so a typo or stale model identifier is obvious to the caller.
*
* @since n.e.x.t

@alewolf alewolf Jun 4, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, but @since n.e.x.t is the project's documented convention for unreleased code — see CONTRIBUTING.md: "Use @since n.e.x.t for new code (will be replaced with actual version on release)." It's used consistently throughout the codebase and gets swapped for the real version at release time, so I've kept it as-is here for consistency.

*
* @param CapabilityEnum $capability The capability the model would be used for.
* @return string The exception message.
*/
private function buildUnmatchedPreferenceMessage(CapabilityEnum $capability): string
{
$requested = [];
foreach ($this->modelPreferenceKeys as $preferenceKey) {
$providerModelParts = explode('::', $preferenceKey, 3);

if ($providerModelParts[0] === 'providerModel' && count($providerModelParts) === 3) {
$requested[] = sprintf('"%s" (provider "%s")', $providerModelParts[2], $providerModelParts[1]);
} elseif ($providerModelParts[0] === 'model' && count($providerModelParts) >= 2) {
$requested[] = sprintf('"%s"', substr($preferenceKey, strlen('model::')));
} else {
// Unexpected key shape; surface the raw value rather than a misleading substring.
$requested[] = sprintf('"%s"', $preferenceKey);
}
}
Comment on lines +1548 to +1559

if ($this->providerIdOrClassName !== null) {
return sprintf(
'None of the requested model preferences (%s) are available from provider "%s" for %s. '
. 'Use a model identifier that the provider exposes, or remove the usingModelPreference() '
. 'call to allow automatic model discovery.',
implode(', ', $requested),
$this->providerIdOrClassName,
$capability->value
);
}

return sprintf(
'None of the requested model preferences (%s) are available from any registered provider for %s. '
. 'Use a model identifier that a provider exposes, or remove the usingModelPreference() '
. 'call to allow automatic model discovery.',
implode(', ', $requested),
$capability->value
);
}

/**
* Parses various input types into a Message with the given role.
*
Expand Down
107 changes: 104 additions & 3 deletions tests/unit/Builders/PromptBuilderTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -902,11 +902,113 @@ public function testUsingModelPreferenceSkipsUnavailableModelId(): void
}

/**
* Tests usingModelPreference falls back to discovery when no preferences are available.
* Tests usingModelPreference throws when none of the requested preferences are available.
*
* Requesting a specific model id that no provider exposes must fail loudly rather than
* silently routing the request to an arbitrary (potentially far more expensive) model.
* See issue #241.
*
* @return void
*/
public function testUsingModelPreferenceThrowsWhenNoPreferenceAvailable(): void
{
$metadata = $this->createTextModelMetadataWithInputSupport('discovered-id');
$providerMetadata = $this->createTestProviderMetadata();
$providerModelsMetadata = new ProviderModelsMetadata($providerMetadata, [$metadata]);

$this->registry->expects($this->once())
->method('findModelsMetadataForSupport')
->with($this->isInstanceOf(ModelRequirements::class))
->willReturn([$providerModelsMetadata]);

// No model must be instantiated when the requested preference is unavailable.
$this->registry->expects($this->never())
->method('getProviderModel');

$this->registry->expects($this->never())
->method('findProviderModelsMetadataForSupport');

$builder = new PromptBuilder($this->registry, 'Test prompt');
$builder->usingModelPreference('unavailable-model');

$this->expectException(InvalidArgumentException::class);
$this->expectExceptionMessage('None of the requested model preferences ("unavailable-model")');

$builder->generateTextResult();
}

/**
* Tests usingModelPreference throws for an unavailable preference when a provider is locked in.
*
* @return void
*/
public function testUsingModelPreferenceThrowsWhenNoPreferenceAvailableForProvider(): void
{
$metadata = $this->createTextModelMetadataWithInputSupport('available-id');

$this->registry->expects($this->once())
->method('getProviderId')
->with('test-provider')
->willReturn('test-provider');

$this->registry->expects($this->once())
->method('findProviderModelsMetadataForSupport')
->with('test-provider', $this->isInstanceOf(ModelRequirements::class))
->willReturn([$metadata]);

$this->registry->expects($this->never())
->method('getProviderModel');

$this->registry->expects($this->never())
->method('findModelsMetadataForSupport');

$builder = new PromptBuilder($this->registry, 'Test prompt');
$builder->usingProvider('test-provider');
$builder->usingModelPreference('unavailable-model');

$this->expectException(InvalidArgumentException::class);
$this->expectExceptionMessage('provider "test-provider"');

$builder->generateTextResult();
}

/**
* Tests usingModelPreference throws and attributes the provider for an unavailable tuple preference.
*
* @return void
*/
public function testUsingModelPreferenceFallsBackToDiscovery(): void
public function testUsingModelPreferenceThrowsWhenTuplePreferenceUnavailable(): void
{
$metadata = $this->createTextModelMetadataWithInputSupport('available-id');
$providerMetadata = $this->createTestProviderMetadata();
$providerModelsMetadata = new ProviderModelsMetadata($providerMetadata, [$metadata]);

$this->registry->expects($this->once())
->method('findModelsMetadataForSupport')
->with($this->isInstanceOf(ModelRequirements::class))
->willReturn([$providerModelsMetadata]);

$this->registry->expects($this->never())
->method('getProviderModel');

$this->registry->expects($this->never())
->method('findProviderModelsMetadataForSupport');

$builder = new PromptBuilder($this->registry, 'Test prompt');
$builder->usingModelPreference(['missing-provider', 'missing-model']);

$this->expectException(InvalidArgumentException::class);
$this->expectExceptionMessage('"missing-model" (provider "missing-provider")');

$builder->generateTextResult();
}

/**
* Tests automatic discovery is still used when no model preference is set.
*
* @return void
*/
public function testFallsBackToDiscoveryWhenNoPreferenceSet(): void
{
$result = $this->createTestResult('Discovered model result');
$metadata = $this->createTextModelMetadataWithInputSupport('discovered-id');
Expand All @@ -929,7 +1031,6 @@ public function testUsingModelPreferenceFallsBackToDiscovery(): void
->method('findProviderModelsMetadataForSupport');

$builder = new PromptBuilder($this->registry, 'Test prompt');
$builder->usingModelPreference('unavailable-model');

$actualResult = $builder->generateTextResult();

Expand Down
Loading