Cost
You pay per token, not per request
Roughly 1 token is about 4 English characters, or 1–2 Chinese characters. Input and output are priced separately, and output usually costs several times more than input.
Repeated prefixes cost less
Several providers discount input tokens that repeat an identical prefix, such as a long system prompt. Keeping that prompt byte-identical between calls can cut input cost noticeably.
Cheap models first
Route easy work to a small model and escalate only when confidence is low. This single change is usually the largest saving available, larger than any negotiation on price.
Limits
Context window is a hard ceiling
The window covers the whole conversation: system prompt, history, retrieved documents and the reply. Exceed it and the request fails outright. Trim history or summarise older turns before you reach it.
Two separate rate limits
Requests per minute and tokens per minute are counted independently, and both are enforced. A burst of short requests can trip the request limit while massive prompts trip the token limit.
Read the status code
401 means a bad key, 429 means a rate limit or empty balance, 400 usually means a malformed body or an oversized context, 5xx is the provider's side. Retry 429 and 5xx with backoff; do not retry 401.
Parameters
Temperature
Controls randomness. Near 0 for extraction, classification and format conversion; 0.7–1.0 for writing and brainstorming. Changing it mid-project makes results harder to compare.
max_tokens
Caps the reply length, not the input. It is still billed as output, so setting it very high does not cost anything by itself, but it does allow long expensive replies.
Streaming
Streaming does not make the model faster. It makes the first characters appear sooner, which is what users actually notice. Use it for anything conversational, skip it for batch jobs.