Skip to content

Commit 67b24f3

Browse files
committed
test: fix generation/conversation e2e (model id -> mistral plain FM, error/null guards)
Two Gen2-cdk e2e suites (generation, conversation) were failing for reasons unrelated to product code. This change fixes the test fixtures and adds fail-fast hygiene guards. Test-side only; no product behavior changes. Model ids The fixtures invoked anthropic.claude-3-haiku-20240307-v1:0, a legacy Bedrock model that has been auto-revoked in the e2e account; invoking it returned ResourceNotFoundException, AppSync resolved null, and the tests crashed on a null-deref. - generation: the @generation transformer's AppSync->Bedrock IAM role grants bedrock:InvokeModel only on the foundation-model ARN, so a cross-region inference profile fails with AccessDenied. The generation fixture is switched to the active on-demand foundation model mistral.mistral-large-2407-v1:0, which the e2e account can invoke under the existing IAM and which satisfies the basic text/scalar assertions. - conversation: routes via @aws-amplify/ai-constructs, which handles inference-profile IAM, so the conversation fixture uses the active inference profile us.anthropic.claude-haiku-4-5-20251001-v1:0. Fail-fast guards generation.test.ts: assert the GraphQL response has no errors and that the generated recipe is not null before dereferencing, so a Bedrock/GraphQL error is printed instead of a TypeError. conversation.test.ts: replace the ineffective toBeDefined() check on the stream part with not.toBeNull(), guard the .length read when the part is null, and add a max-iteration / overall-timeout guard inside the subscription for-await loop so a non-streaming assistant fails fast with a descriptive message instead of cascading into the jest timeout. --- Prompt: Split the mixed e2e/CI branch into a clean draft PR off main containing only the generation/conversation e2e test fixes: switch the dead legacy claude-3-haiku model id to a working model (generation -> mistral plain on-demand FM, conversation -> working inference profile) and add the error/null fail-fast guards. tsc the construct-tests package and commit (no --no-verify).
1 parent 9a2c742 commit 67b24f3

4 files changed

Lines changed: 30 additions & 8 deletions

File tree

packages/amplify-graphql-api-construct-tests/src/__tests__/conversations/conversation.test.ts

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -111,14 +111,34 @@ describe('conversation', () => {
111111
expect(message.conversationId).toEqual(conversationId);
112112

113113
const events: AmplifyAIConversationMessageStreamPart[] = [];
114+
// Guard against a non-streaming assistant: fail fast with a clear message instead of
115+
// hanging on the subscription until the surrounding 20-minute jest timeout cascades.
116+
const MAX_STREAM_EVENTS = 1000;
117+
const streamDeadline = Date.now() + ONE_MINUTE;
114118
// expect to receive the assistant response in the subscription
115119
for await (const event of subscription) {
116-
events.push(event.onCreateAssistantResponsePirateChat);
120+
if (Date.now() > streamDeadline) {
121+
throw new Error(
122+
`Timed out waiting for a streamed assistant response with a stopReason after ${ONE_MINUTE}ms ` +
123+
`(received ${events.length} events). The assistant may not be streaming a response.`,
124+
);
125+
}
126+
if (events.length >= MAX_STREAM_EVENTS) {
127+
throw new Error(
128+
`Received ${events.length} stream events without a stopReason; aborting to avoid an unbounded loop. ` +
129+
`The assistant may not be terminating its response.`,
130+
);
131+
}
132+
133+
const streamPart = event.onCreateAssistantResponsePirateChat;
134+
events.push(streamPart);
117135
// expect event to contain `p`
118-
expect(event.onCreateAssistantResponsePirateChat.p).toBeDefined();
119-
expect(event.onCreateAssistantResponsePirateChat.p.length).toBeGreaterThanOrEqual(0);
136+
expect(streamPart.p).not.toBeNull();
137+
if (streamPart.p != null) {
138+
expect(streamPart.p.length).toBeGreaterThanOrEqual(0);
139+
}
120140

121-
if (event.onCreateAssistantResponsePirateChat.stopReason) break;
141+
if (streamPart.stopReason) break;
122142
}
123143
const accumulatedP = events.map((messageStreamPart) => messageStreamPart.p).join('');
124144
expect(accumulatedP.length).toBeGreaterThan(0);

packages/amplify-graphql-api-construct-tests/src/__tests__/conversations/graphql/schema-conversation.graphql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ type Mutation {
66
toolConfiguration: AmplifyAIToolConfigurationInput
77
): AmplifyAIConversationMessage
88
@conversation(
9-
aiModel: "anthropic.claude-3-haiku-20240307-v1:0"
9+
aiModel: "us.anthropic.claude-haiku-4-5-20251001-v1:0"
1010
systemPrompt: "You are a helpful chatbot that responds in the voice and tone of a pirate. Respond in 20 words or less."
1111
auth: { strategy: owner, provider: userPools }
1212
)

packages/amplify-graphql-api-construct-tests/src/__tests__/generations/generation.test.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,9 @@ describe('generation', () => {
6363
};
6464

6565
const generateRecipeResult = await doAppSyncGraphqlQuery({ ...args, query: generateRecipe, variables });
66+
expect(generateRecipeResult.body.errors).toBeUndefined();
6667
const recipe = generateRecipeResult.body.data.generateRecipe;
68+
expect(recipe).not.toBeNull();
6769
expect(recipe.name).toBeDefined();
6870
});
6971
});

packages/amplify-graphql-api-construct-tests/src/__tests__/generations/graphql/schema-generation.graphql

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,14 @@ type Recipe {
66
type Query {
77
summarize(input: String): String
88
@generation(
9-
aiModel: "anthropic.claude-3-haiku-20240307-v1:0"
9+
aiModel: "mistral.mistral-large-2407-v1:0"
1010
systemPrompt: "summarize the input."
1111
inferenceConfiguration: { temperature: 0.5 }
1212
)
1313

1414
generateRecipe(description: String): Recipe
15-
@generation(aiModel: "anthropic.claude-3-haiku-20240307-v1:0", systemPrompt: "You are a 3 star michelin chef that generates recipes.")
15+
@generation(aiModel: "mistral.mistral-large-2407-v1:0", systemPrompt: "You are a 3 star michelin chef that generates recipes.")
1616

1717
solveEquation(equation: String): Int
18-
@generation(aiModel: "anthropic.claude-3-haiku-20240307-v1:0", systemPrompt: "Solve the equation and return the result.")
18+
@generation(aiModel: "mistral.mistral-large-2407-v1:0", systemPrompt: "Solve the equation and return the result.")
1919
}

0 commit comments

Comments
 (0)