Read the upstream page: AWS Bedrock, vendor keys and HIPAA routes
As a Workforce Intelligence operator, read the Upstream keys and models page: the AWS Bedrock connection and its price check, each vendor key and its model listing, open circuits, and the routes that may serve a HIPAA account.
The Upstream keys and models page shows how the service reaches the models it offers: the AWS
Bedrock connection, each vendor key by its id, what each key and Bedrock can serve, and the
circuits that are open right now. It is read only. Every view of the page is recorded in the
operator log as upstream.view. The page never shows a key, an AWS account id or an ARN.
To open it, select Upstream keys and models in the operator menu, or select Keys and models on the Upstream health card of the operator Dashboard.
Upstream health on the dashboard#
The Upstream health card lists every provider with its Active keys and Open circuits.
AWS Bedrock is listed as bedrock, with in use (AWS role) while its connection is connected
or its first check is still running, and not in use (AWS role) otherwise. On a development
stack that signs Bedrock calls with dev credentials, the parentheses name them instead (see
below). A provider needs
attention when it has no active key (for Bedrock: it is not in use) or has an open circuit, and the
card says how many need attention.
AWS Bedrock#
The service calls AWS Bedrock with its own AWS role (the ECS task role in production), so Bedrock
has no vendor key in the key document WICENTRAL_UPSTREAM_KEYS. The AWS Bedrock card shows:
| Row | What it means |
|---|---|
| Credentials | Who signs Bedrock calls. AWS role in production. On a development stack, Dev credentials (profile name), Dev credentials (access key ...last 4) or Dev credentials (API key), followed by the entry's id. The page never shows a secret, a token or a full access key id. |
| Connection | connected: the last listing of the home region succeeded. checking: the startup check has not finished; Bedrock routes are used meanwhile, and a route that cannot work fails on its first call and opens that model's circuit. unavailable: the listing failed and no recent listing is trusted; Bedrock routes are not used. off: Bedrock is switched off. |
| Region | The home region, WICENTRAL_BEDROCK_REGION. |
| Regions listed | The home region, then every region a catalogue model is pinned to (Stability image models run in us-west-2). |
| Catalogue models listed | How many of the catalogue's enabled Bedrock routes the listings include, out of all of them. Routes under Not enabled are not counted. Unknown until the connection is connected. |
| Models Bedrock lists | Every active model and system inference profile the listings returned. |
| Last listed | Minutes since the last good listing of the home region, or Never. |
When the connection is unavailable, the card names the last check's failure, for example
credentials (no AWS credentials, or they were refused or expired), sso_session_expired or
profile_not_found (a development stack's dev profile cannot be resolved), auth with the AWS error
code AccessDeniedException (the role lacks the list permissions), rate_limited, http_error,
timeout, unreachable, bad_response, or ambient_auth_env (one of
AWS_ROLE_NAME, AWS_PROFILE_NAME or AWS_WEB_IDENTITY_TOKEN is set, which would make calls sign
as a different identity than the one checked). The check is retried, starting after a minute and
backing off up to the model-discovery period, so a fixed IAM policy is picked up without a restart.
Catalogue models not listed names any catalogue Bedrock model the listings do not include. That model is skipped until a later listing includes it. A model can be listed but not enabled for the account: its first call is refused, which opens a circuit for that model only.
Open circuits lists each Bedrock model whose circuit is open, with its Reason, its Upstream error and Cooldown left. Bedrock circuits are scoped to one model, because Bedrock access is granted per model. "every Bedrock model" means Bedrock's credentials as a whole failed, for example expired ones.
The Reason says why the circuit opened:
| Reason | Meaning |
|---|---|
| model_unavailable | Bedrock refused this model for the account (not enabled, no access, or not found). It stays closed to calls for at least five minutes, longer if it keeps failing. |
| auth_failed | Bedrock's credentials failed (missing, invalid or expired). Shown on "every Bedrock model". |
| backoff | The model was throttled, overloaded or timed out. It is retried after a short wait. |
| retry_after | The upstream asked central to wait a given time. |
The Upstream error is the code Bedrock gave and the first 200 characters of its message, for
example access_denied: "openai.gpt-6-luna is not available for this account. You can explore
other available models on Amazon Bedrock. ...". accessdeniedexception is the same refusal on a
Claude model. Credentials are removed from the message, and no message is shown for a refused
request (an HTTP 400), which could repeat what the caller sent. "Not reported" means Bedrock gave no
code or message.
AWS Bedrock is off when WICENTRAL_BEDROCK=off, or when the service runs with the fake upstream
driver. Bedrock routes are then never used.
Not enabled#
The Not enabled list names, by model id, every Bedrock route the model catalogue
(central/src/wi_central/routing/data/catalogue.json) switches off with "enabled": false. Such a
route stays in the catalogue and the price table (the price check above still covers it), but it is
never offered, never called, not counted under Catalogue models listed, never counts as a
HIPAA-eligible route, and the task role has no IAM grant for it.
The catalogue enables every route, so the card normally shows Every Bedrock route in the catalogue is enabled. in place of the list. The list appears only when a reviewed change switches a route off, for example because the AWS account cannot call that model.
To switch a route off, set "enabled": false on it in a reviewed change to the catalogue; to enable
it again, remove the flag (or set it to true). Merging the change deploys the infrastructure stack,
which removes or adds the route's IAM grant, and the service, which stops or starts offering it.
Enable a Bedrock route only once the AWS account can call that model: a call the account cannot
make fails with accessdeniedexception and opens that model's circuit.
The catalogue lists one version of each Claude model, Claude 5.5, through Anthropic's API and
through Bedrock. When every route of Claude 5.5 fails, is refused by AWS or is switched off, a
claude- name is answered by the equivalent model of another maker at the same level (for
example GPT on Bedrock), never by an older Claude. When a Claude 5.5 route is usable again, it is
used first, with no restart.
Dev credentials on a development stack#
A development stack has no AWS role, so its key document may hold a bedrock list with one active
entry that names the developer's own AWS identity, in exactly one form: a named AWS profile
(profile, for example an IAM Identity Center profile), access keys (access_key_id and
secret_access_key, plus session_token for temporary keys), or an Amazon Bedrock API key
(api_key, generated in the Bedrock console under API keys; a short-term key expires within
12 hours and works only in the region it was generated in). Only Bedrock calls use it. The entry
id becomes the id in Bedrock's circuits. When a profile's sign-in session expires, the connection
turns unavailable with sso_session_expired until the developer signs in again.
Production never uses dev credentials: the service refuses to start when its key document has a
bedrock list, and rejects a later document that has one (ERROR
upstream_keypool_reload_rejected), keeping the previous key set in service.
What a missing Bedrock connection disables#
While Bedrock is unavailable or off, only Bedrock routes stop. Calls fall back to vendor keys where the catalogue has one:
claude-haiku,claude-sonnetandclaude-opusare served only through an Anthropic key, thegpt-names only through an OpenAI key, andgrokonly through an xAI key (or, for any of them, the equivalent model of the same level from another maker that has a key).- The
reasoning-,vision-,embedding-andimage-names are served by the other vendors that have a key. A Knowledge Base whose stored embedding model is served only through Bedrock cannot index or search until Bedrock connects. - HIPAA accounts are served only through vendor keys marked
hipaa_eligible. If one of chat, embeddings or image generation has no such key, HIPAA mode cannot be turned on for any account. - With no vendor key either, every call fails with HTTP 503 and the service logs
upstream_keypool_emptyas an alert.
Price check#
At startup the service reads the AWS Price List once and compares it with the Bedrock rows of its price table. The result appears under Price check on the AWS Bedrock card:
- confirmed: every Bedrock price matches, with the number of models checked and the time (UTC).
- differs: a table lists each difference by Model and Charge (for example
input,cache_read,long_context.inputorper_image), with the Price table amount and the AWS Price List amount in exact dollars per million tokens or per image. "no charge" means that side has no such charge. - unavailable: the Price List could not be read; the card names the error class only.
- checking: the check has not finished.
- off: the check is switched off (
WICENTRAL_BEDROCK_PRICE_CHECK=off), or Bedrock is off.
The check never changes a price. Calls are always billed and costed at the price table. When the
check reports differs, update the price table with central/scripts/fetch_bedrock_prices.py
and deploy it.
HIPAA-eligible routes#
The HIPAA-eligible routes card shows what may serve a HIPAA account (see Operate HIPAA accounts):
- AWS Bedrock: Eligible, in use, Eligible, not connected or Off. Every Bedrock route is HIPAA eligible.
- Vendor keys marked hipaa_eligible: each provider with the ids of its active keys marked
"hipaa_eligible": true, or None. - HIPAA mode can be turned on: Yes, or No with the capabilities (chat and reasoning, embeddings, image generation) that have no eligible route right now.
Vendor keys#
Each provider in the key document has a card with one row per key: Key (its id), State
(active, draining or disabled), Model listing (ok, failed with the reason, or unknown), Last
listed, Listed, Catalogue matched and Catalogue models missing. Below it, Open
circuits lists each open circuit by Key, Scope (the whole key, or one model), Reason,
Upstream error (the vendor's short error code, such as model_not_found; vendor messages are
never shown) and Cooldown left.
When the key document has no keys, the No upstream keys card says so. With Bedrock connected
or checking, calls are served through AWS Bedrock's enabled routes: the claude- names by Claude
5.5, the gpt- names by the GPT model of the same name and grok by Grok 4.7, each on Bedrock;
the gemini- names (Bedrock has no Gemini) by the cheapest other maker's model of the same level
(gemini-flash-lite by Claude Haiku 5.5, gemini-flash by GPT-6.1 Terra, gemini-argon by
GPT-6.1 Sol); the reasoning- and vision- names by the cheapest of their best-ranked Bedrock
candidates; the embedding- names by Cohere Embed v4 and the image- names by Stability. A
route under Not enabled never serves. Without Bedrock, every call fails
until a key is added or Bedrock connects.