Skip to content

Commit 227d418

Browse files
authored
[Bug] Fix (Azure) OpenAI streaming (#1288)
Getting some responses from OpenAI with no choices, which we did not handle gracefully. Also, `prompt_token_details` is not present in Azure.
1 parent dc89519 commit 227d418

1 file changed

Lines changed: 5 additions & 6 deletions

File tree

guidance/models/_openai_base.py

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -281,15 +281,14 @@ def _handle_stream(self, chunks: Iterator["ChatCompletionChunk"]) -> Iterator[Ou
281281
usage.input_tokens += chunk.usage.prompt_tokens
282282
# Estimate forward passes as number of completion tokens
283283
usage.forward_passes += chunk.usage.completion_tokens
284-
if chunk.usage.prompt_tokens_details is not None:
284+
if getattr(chunk.usage, "prompt_tokens_details", None) is not None:
285285
if chunk.usage.prompt_tokens_details.cached_tokens is not None:
286286
usage.cached_input_tokens += chunk.usage.prompt_tokens_details.cached_tokens
287-
try:
288-
choice = chunk.choices[0]
289-
except IndexError:
290-
# TODO: azure seems to return empty choices sometimes (on first chunk?)
291-
# Need to make this more robust
287+
if chunk.choices is None or len(chunk.choices) == 0:
288+
# Azure seems to return empty choices sometimes (on first chunk?)
289+
# OpenAI seems to return None choices sometimes (after giving usage?) (for audio only?)
292290
continue
291+
choice = chunk.choices[0]
293292
delta = choice.delta
294293
if delta.content is not None:
295294
assert audio is None

0 commit comments

Comments
 (0)