SageMaker vs Bedrock vs Direct APIs: Choosing AI Architecture on AWS

Key Takeaways

• For most AI features in a product on AWS, the real choice is Amazon Bedrock or a direct provider API. Amazon SageMaker AI fits once you need to customize, own or host the model.

• Bedrock now serves both Anthropic's Claude and OpenAI's GPT models, with APIs compatible with each provider's own, so choosing it over a direct API comes down to data terms, contracts and how fast each model version arrives.

• For the same open-weight model (gpt-oss-120b) at AWS list prices in November 2026, Bedrock costs less than 2 always-on SageMaker AI instances up to a theoretical crossover of about 15 million requests a month.

• Some Bedrock models, including Claude Fable 5, need a data retention mode that a zero-retention setting blocks, so check the setting in every Region before launch.

• Send every model call through one layer in your own code to keep a later move between Bedrock and a direct API small, but check which APIs and features each model supports on each endpoint.

Your product already runs on AWS, and its AI feature works in a prototype with an OpenAI or Anthropic key. Since mid-2026, Amazon Bedrock has also offered OpenAI's GPT models alongside Claude, so choosing where that call runs in production now means weighing SageMaker vs Bedrock vs the direct provider API you already use.

For most AI features, the decision that matters is Bedrock or a direct API, and SageMaker AI becomes worth the extra operational work once you need to change, own or host the model.

We'll compare the 3 routes by cost, operational effort and data handling, then work through a break-even calculation and a decision table that lists what we would not build.

The comparison uses AWS documentation and pricing checked on 3 November 2026, alongside tests from the AI assistant we run on our website.

What Is the Difference Between Bedrock and SageMaker?

Amazon Bedrock is a managed, serverless service for using ready-made foundation models through one API, and Amazon SageMaker AI is a managed service for building, customizing, training and hosting the models you choose.

AWS's decision guide (updated 23 July 2026) points to SageMaker AI when you need to change a model's behavior or manage its cost, throughput and latency yourself.

A direct provider API is the third route: your own services call OpenAI or Anthropic with the provider's key, outside AWS's AI services.

Here's SageMaker vs Bedrock vs a direct provider API side by side, from AWS's decision guide and the Bedrock pricing page (3 November 2026):

Running open-weight models yourself on Amazon EC2 or EKS needs a machine learning (ML) operations team of your own and is outside this guide.

What Amazon Bedrock Gives You

Amazon Bedrock gives you models from many providers through one API and one AWS bill, with no servers to manage. Bedrock adds Knowledge Bases for retrieval, Guardrails for filtering and AgentCore for agents. AWS says an existing OpenAI app moves to its OpenAI-compatible endpoints by changing the base URL and API key, though support varies by model and endpoint: Claude uses an Anthropic-compatible Messages API instead.

What Amazon SageMaker AI Gives You

Amazon SageMaker AI gives you control over the model itself: you choose it, customize it and run it on instances you pick. If you're weighing SageMaker JumpStart vs Bedrock for a ready-made model, JumpStart offers nearly 1,000 models, but AWS lists Claude in Bedrock only. Serverless customization of open-weight models runs through a guided interface that AWS says needs no ML expertise, and real-time endpoints can scale to zero when idle.

Bedrock vs a Direct OpenAI or Anthropic API

Calling OpenAI or Anthropic directly from your AWS services is a valid production route. You get the provider's own contract, data terms and new features as they ship, and your prompts go to the provider, outside your AWS account's controls. With both providers' models on Bedrock, the choice comes down to 4 things:

• Billing: one AWS bill, or a second vendor with its own contract.

• Data handling: AWS's terms, or the provider's.

• Credentials: AWS Identity and Access Management (IAM) roles, or provider keys you store and rotate.

• Model versions: when the exact version and feature you need reach each route, which you check per model.

When to Use Bedrock, SageMaker AI or a Direct API

Pick the route by what you're building: assistants, agents and RAG features usually start on Bedrock or a direct API, and SageMaker AI is worth its extra work for customized models and steady, high-volume inference.

AI Assistants, Chatbots and Agents

For assistants, chatbots and agents, start on Bedrock or the provider's own API, and choose by data terms and the features you need. Our guide to custom AI chatbot development covers the build itself.

RAG Over Your Own Documents

For retrieval-augmented generation (RAG), where the model answers from documents retrieved for each question, Bedrock Knowledge Bases suit a document collection in Amazon S3. When answers must combine documents with live product data, we'd keep the vectors next to that data, for example with pgvector on Amazon RDS or Aurora. Here's how we approach RAG over private documents.

Customizing a Model's Behavior

Customize a model only after you've measured where prompting fails, because both services need training data and a fixed evaluation set. Bedrock supports fine-tuning, distillation and reinforcement fine-tuning for supported models. SageMaker AI adds serverless customization of open-weight models and full training jobs.

Using Bedrock and SageMaker Together With Custom Model Import

Bedrock and SageMaker AI work together through Custom Model Import, which lets Bedrock serve a model you customized elsewhere. In the path AWS describes, you start on Bedrock with prompts and RAG, customize with SageMaker AI when prompting isn't enough, and serve the result on either service.

Custom Model Import accepts Hugging Face weights for supported architectures, including Llama, Mistral, Qwen and GPT-OSS, in a limited set of Regions. According to the Bedrock pricing page, an idle imported model scales to zero.

If you can't name the training data and the evaluation set you'd customize with, start on Bedrock or a direct API.

Adding an AI feature to a product on AWS?

We add AI to products you already run, with the model layer, guardrails and cost limits designed in.

Explore AI integration services

Bedrock vs SageMaker Cost: What Determines the Price

Bedrock and direct APIs charge per token, so the bill follows your traffic. A SageMaker AI endpoint charges per instance-hour while it runs, so it costs less only when it stays busy.

On AWS's Bedrock pricing page (checked 3 November 2026), batch inference is 50% below on-demand, the Flex tier is 50% below Standard, and Priority costs 75% more. The SageMaker AI pricing page bills an endpoint for every hour its instances run. Machine Learning Savings Plans cut that rate for a 1- or 3-year commitment, by up to 64% according to AWS's decision guide.

Worked Example: The Same Open-Weight Model on Bedrock and SageMaker AI

For the same open-weight model at AWS list prices, Bedrock on demand costs less than 2 always-on SageMaker AI instances up to a theoretical crossover of about 15 million requests a month. The example uses gpt-oss-120b, OpenAI's open-weight model, because Bedrock and SageMaker JumpStart both offer it.

The assumptions:

• Prices: AWS list prices for US East (N. Virginia) on 3 November 2026. Bedrock's Standard tier charges $0.15 per million input tokens and $0.60 per million output tokens. SageMaker AI hosting costs $4.2039 an hour on ml.g7e.2xlarge, the instance JumpStart deploys this model to by default.

• Workload: 1,500 input and 300 output tokens per request. The 300 include the model's reasoning tokens, which bill as output. Reasoning is on by default for gpt-oss-120b, and we assume the low setting.

• Hosting: 2 instances for redundancy, running 730 hours a month, before Savings Plans, Flex or batch discounts.

The formulas:

• Bedrock monthly cost = requests × (input tokens × input price + output tokens × output price), or $0.000405 per request here.

• SageMaker AI monthly cost = hourly rate × hours × instances, or $6,137.69 here.

• Crossover = SageMaker AI monthly cost ÷ Bedrock cost of one request.

Bedrock costs less up to about 15.2 million requests a month, or about 500,000 a day:

A Savings Plan lowers the crossover in exchange for a commitment. For this instance, AWS lists $3.3106 an hour on a 1-year plan with no upfront payment (21% off) and $2.2247 on a 3-year plan paid upfront (47% off). That moves the crossover to about 11.9 million and 8.0 million requests a month. In the chart, find your own monthly volume: below 8 million requests, Bedrock costs less on any of these plans.

The crossover is theoretical. It holds only if each instance can sustain about 2.9 requests a second around the clock within your latency target, or about 865 generated and 4,300 input tokens a second. AWS's G7e launch post says a single G7e.2xlarge can host gpt-oss-120b, so the model fits, but the post gives no throughput figure and we haven't load-tested this setup. If an instance can't sustain that rate, SageMaker AI costs more per request than Bedrock at every volume, at these prices. SageMaker AI's inference recommendations can run that test on up to 3 instance types with your own data and concurrency.

The calculation leaves out the engineering time to run an endpoint. AWS also doesn't document how Bedrock serves the model, so compare both routes' answers and latency first.

Bedrock vs a Direct API for the Same Model

When the same model is on Bedrock and its provider's API, compare the same version, token volumes and pricing conditions (tier, batch, caching, committed spend) on both pricing pages on the same day, then test both routes with your own cases.

Costs That Don't Appear on a Price List

Every route has costs no price list shows: idle capacity, engineers on call, evaluation runs when a model or prompt changes, logging, retries, and re-embedding documents after an embedding model change. Put a budget in code, per request, per user and per day. Choosing a cheaper model is a quality decision first: test it on your own cases before comparing bills.

Security, Data Privacy and HIPAA: What Each Route Does With Your Data

On Bedrock, model providers have no access to your prompts and completions, and on a SageMaker AI endpoint the model runs for your account alone. A direct provider API sends prompts to the provider under its own terms, so the security decision starts with where your data is allowed to go.

How Bedrock Data Retention Affects Which Models You Can Use

Bedrock data retention is a setting per Region, and it decides which models you can call. Claude Fable 5 and 5.1, for example, need a mode in which AWS keeps prompts and completions inside AWS for up to 30 days for review. Those models are unavailable in Regions configured for zero retention. With cross-region inference, retained data is stored in the Region that processes the request. AWS's data retention page lists the modes and how to set them.

Authentication and Private Networking

Bedrock supports IAM-based authentication with temporary AWS credentials, while direct provider APIs generally require separate provider-issued credentials that must be securely managed. Bedrock's own API keys come in 2 kinds: short-term keys, valid for up to 12 hours, which AWS recommends for production, and long-term keys for exploration only. AWS PrivateLink keeps traffic between your virtual private cloud (VPC) and Bedrock off the public internet.

What Direct Provider APIs Do With Your Data

A direct provider API puts your data under the provider's own terms. OpenAI says it doesn't train on API data unless you opt in, keeps abuse-monitoring logs for up to 30 days by default and offers zero data retention on approval. Anthropic says it doesn't train on inputs and outputs from its commercial products by default. The provider's data processing agreement applies too.

Are Bedrock and SageMaker AI HIPAA Eligible?

Yes: AWS lists both Amazon Bedrock and Amazon SageMaker AI as HIPAA eligible services (updated 15 October 2026). You can use them for protected health information (PHI) under a business associate agreement (BAA) with AWS, but compliance still depends on your application. A direct API needs that provider's own BAA.

Before launch, read the retention setting in each Region you use, and confirm every model you call is available under it.

How to Choose the Right AI Architecture on AWS

Choose by your constraints: where your data may go, which model the feature needs, how steady your traffic is, who will own the model and how cheaply you can switch later.

5 Questions That Decide the Route

Answer these 5 questions before writing code:

1. Where may the data go? Contracts, a BAA and residency rules decide which providers may receive your prompts.

2. Which model and features do you need? Check that the exact version is on the route you pick.

3. How steady is your traffic? Per-token billing suits uneven traffic; an endpoint pays off only when busy.

4. Who will own the model? Customization brings training data, evaluation runs and on-call work.

5. How cheaply can you switch? One model layer in your code and short commitments keep the next move cheap.

If one provider's contract and data terms meet your customers' requirements, its API alone is enough, with no need to add Bedrock just to have another route. Keep the calls behind one model layer so you can switch later if those requirements change.

Decision Table: Where to Start and What We Would Not Build

Find the row that matches your situation; the last column is what we would not build.

How We Test the Options Before Choosing

We test the options by turning the feature's requirements into a fixed set of test cases, running every candidate model and route on the same cases, and comparing answer quality, latency and cost together.

When we chose the model and route for the AI assistant on our website in June 2026, we tested 2 candidate models against its real system prompt using 14 scenarios twice per model. Eight of the faster model's 28 responses contained hallucinations, compared with 16 of the other model's 28.

In local tests, calling Google's API directly was 9–15% faster per round trip than going through an AI gateway. Every model call goes through one layer in our code, so changing the route was a setting, and so is changing it back.

Those 56 responses were enough to choose between 2 candidates for that assistant, not to establish which model performs better overall. The assistant runs on Vercel and Google, not AWS, so what carries over is the method. Our earlier benchmark of 10 models for the same assistant applied it to latency and tool calling.

Whichever row of the decision table fits, record the reason and keep the test cases, so you can rerun the comparison when a new model ships.

Next Steps for Choosing AI Architecture on AWS

Next, answer the 5 questions for your feature and test the candidate routes on the same cases before you commit.

DBB Software has been an AWS Select Tier Partner since 2024, with ISO/IEC 27001-certified practices. On a healthcare platform we support on AWS, services authenticate with IAM roles instead of stored long-lived keys, and patient records remain stored in each tenant's AWS account.

If you're deciding where an AI feature in your AWS product should run, book a 30-minute architecture call to compare the options against your requirements. If you're still defining the feature, get a free scope document first.

FAQ