← Back to naveendavuluri.com A beginner's guide to AI chatbots

How AI Chatbots Work — Explained Like I'd Tell a Friend

Words like "tokens," "context windows," and "agents" often get used as if everyone already knows what they mean. This guide explains the ideas behind modern AI chatbots in plain language—no computer science degree required.

The whole story in about 60 seconds

An AI chatbot combines a language-producing engine with the app you interact with. Here is the basic journey from question to answer:

  1. Your prompt and instructions enter the application.
  2. The text is split into small pieces called tokens.
  3. The language engine connects each piece with the earlier words and instructions that matter.
  4. The app may retrieve documents or use approved tools.
  5. The language engine generates a response one token at a time.
  6. You verify important claims and decide what happens next.
Path 1

Start here

The four ideas every AI user should understand first.

Start here

The Model and the Assistant Are Not the Same Thing

The model creates language. The surrounding app supplies the interface, memory, documents, tools, and safety controls.

A large language model, or LLM, is trained on large amounts of text so it can learn patterns in language. Training is the long process that creates its learned settings. Using the finished model to answer a prompt is called inference.

The chatbot you open is more than that model. Its application may remember parts of the conversation, search documents, browse the web, call tools, or ask for approval. Those abilities come from the whole system—not from the language model alone.

Language model Predicts and generates
+
AI application Context, tools, permissions
=
AI assistant The experience you use
A chatbot is usually a language model wrapped in an application.
Why it matters: This distinction explains why two products using similar models can behave very differently.
Think of it like: the model is the engine; the AI application is the entire car, including the steering wheel, brakes, navigation, and rules about who may drive it.
Back to contents →
Start here

Tokens

The small pieces of text a language model processes.

Language models do not receive sentences exactly as we see them. The chatbot first splits text into pieces called tokens. A token can be a whole word, part of a word, punctuation, or another recurring text pattern. The word unbelievable, for example, might be split into several pieces—but the exact split depends on the model.

Models also produce answers token by token. That is why answers appear gradually and why services often measure limits and usage in tokens rather than words.

One word, three tokens — the pieces AI actually processes.
Why it matters: Tokens affect how much material fits into a conversation, how long an answer can be, and sometimes how usage is priced.
Think of it like: Lego bricks. We see the finished sentence; the model works with the smaller pieces that can be assembled into it.
Back to contents →
Start here

Context Window

How much information the model can consider for one response.

The context window includes the current prompt, relevant earlier messages, instructions, and any documents or tool results supplied to the model. It is measured in tokens and has a fixed maximum size.

When a conversation grows beyond that limit, the application must decide what to keep. It may remove older material, summarize it, or retrieve relevant details again. Different AI products handle this differently.

The window has a fixed size, so the application must remove, summarize, or retrieve information when it fills up.
Why it matters: A long conversation can lose important details. Restating key requirements or starting a focused new conversation can improve results.
Think of it like: a whiteboard with limited space. Once it fills up, something must be erased, condensed, or stored elsewhere before new material can be added.
Common misconception: A long context window is not the same as perfect memory. The model can overlook information even when it technically fits.
Back to contents →
Start here

Hallucinations

When an AI produces plausible language that is unsupported, incorrect, or invented.

A language model is designed to produce a likely continuation—not to guarantee that every statement is true. It can invent a statistic, citation, event, explanation, or source while sounding completely confident.

Hallucinations are not limited to obscure questions, and confidence is not evidence. Verify important medical, legal, financial, safety, and factual claims with trustworthy sources.

Fluent and confident-sounding is not the same thing as accurate.
Why it matters: The right way to use AI is not blind trust or total distrust. Use it for assistance, then apply verification in proportion to the stakes.
Think of it like: a fluent storyteller who fills a missing detail smoothly instead of stopping to say, "I do not know."
Common misconception: Telling a model not to hallucinate cannot eliminate the problem.
Back to contents →
Path 2

Using AI in practice

How applications add knowledge, tools, repeatable methods, and multi-step work.

In practice

RAG (Retrieval-Augmented Generation)

Looking up relevant material before composing an answer.

With RAG, an application searches a selected collection—such as company policies, product manuals, or support articles—before asking the model to answer. The retrieved passages are added to the model's context so it can use information that was not stored reliably in its learned settings.

Many RAG systems create numerical "meaning fingerprints" for passages—these are called embeddings. They help the system find related passages even when the wording is different. Embeddings are explained in more detail in the optional "Under the hood" section. Good retrieval can make answers more current and traceable, but poor sources or the wrong passage can still produce a bad answer.

Search the highlighted, trustworthy passage first — then answer from what was actually found.
Why it matters: RAG is useful when answers should be grounded in a specific, updateable source of truth.
Think of it like: an open-book exam: the student first finds the relevant page, then writes an answer using what is printed there.
Common misconception: RAG can reduce unsupported answers, but it does not guarantee accuracy.
Back to contents →
In practice

MCP (Model Context Protocol)

A standard way for compatible AI applications to connect to tools and data sources.

A model by itself cannot open your calendar, read a company file, or update a project board. An AI application needs a connector that describes the available tools and how to call them. MCP provides a shared set of connection rules—a protocol—for doing that.

Compatibility is only one part of the story. The connector, application, permissions, and approval flow determine what the assistant may actually access or change. MCP does not automatically make every connection safe or make every tool work with every AI product.

One shared connector shape instead of a custom cable for every single app.
Why it matters: A common protocol can reduce the need to build a completely different integration for every combination of assistant and service.
Think of it like: a widely used connector standard. It gives compatible systems a shared plug shape, while access controls still determine whether power is allowed to flow.
Common misconception: MCP is the connection standard—not the model, the tool itself, or an automatic security guarantee.
Back to contents →
In practice

Agents

AI systems that can plan and perform several steps toward a goal.

An agent repeatedly works through a simple cycle: understand the goal, choose a next step, use an allowed tool, inspect the result, and decide what to do next. That can support work such as researching several sources, organizing files, or writing and testing code.

Agents can still misunderstand the task, choose a poor step, fail to finish, or require human approval. Good agent systems limit permissions, record what happened, and pause before consequential actions.

An agent can take several steps, checking its progress and pausing when it needs help or approval.
Why it matters: Agents shift AI from producing a single answer toward helping complete a bounded workflow.
Think of it like: giving an assistant a goal, a checklist, and access to only the tools needed—not handing over unlimited authority.
Common misconception: More autonomy does not automatically mean more reliability.
Back to contents →
In practice

Skills

Reusable playbooks that help an AI assistant handle a particular task consistently.

In some AI products, a skill is a package of instructions, examples, templates, and sometimes supporting scripts for a specific kind of work. A report-writing skill might define the expected sections, tone, review checks, and output format.

"Skill" is not one universal technical standard; products may use different names and packaging. The underlying idea is consistent: give the assistant a repeatable method instead of making it improvise every step.

A written playbook the AI follows every time, instead of improvising from scratch.
Why it matters: Well-designed skills make results more consistent and can preserve useful team practices.
Think of it like: a clear instruction card given to a capable new employee for a task your team performs regularly.
Back to contents →
In practice

Local and Smaller Models

Models designed to run on ordinary devices or use fewer computing resources.

A local model runs on your own computer or phone instead of sending every prompt to a remote model service. When both the model and its surrounding application operate fully offline, prompts and responses can remain on the device. Some local applications still use cloud tools, updates, or send usage data, so check the product's settings and documentation.

Smaller models are easier to run locally. Quantization stores learned numbers with less precision. Distillation trains a smaller student model with guidance or examples from a larger teacher, often alongside other data. Pruning removes parts that contribute little under the chosen method.

In a fully offline setup, the model and your prompts can remain on your device.

How models are made smaller

Much of the model's capability, packed into a smaller, faster, cheaper-to-run file.
Why it matters: Local operation can improve privacy, offline access, control, and cost—but often trades away some speed, capability, or convenience.
Think of it like: keeping a capable reference guide on your device instead of calling a remote expert for every question.
Common misconception: "Local model" does not automatically mean the entire app is private or offline.
Back to contents →
Path 3 · Optional

Under the hood

A little more detail for readers curious about what happens inside the model.

Under the hood

Token IDs and Embeddings

Turning pieces of text into number-based representations the model can calculate with.

Each token is assigned an integer called a token ID. The ID is an address in the model's list of known tokens, not a measure of meaning. The system uses that address to look up an embedding: a list of numbers that the model can process.

Initial embeddings capture learned relationships between tokens. Later processing layers then build richer representations based on the surrounding text. The representation of "bank" can therefore develop differently in "river bank" and "savings bank." Most internal computation happens on these lists of numbers—not directly on the token ID.

Within one model, a token ID is the lookup address used to retrieve that token's embedding.

From an ID to a meaning-rich representation

An embedding is a token's coordinates on a giant map of meaning.
Why it matters: This is how text enters a mathematical system without pretending that the ID number itself means anything.
Think of it like: a library call number that leads to a detailed record. The call number finds the entry; the record contains the useful description.
Common misconception: A token's starting embedding is not a complete, fixed definition of the word in every sentence.
Back to contents →
Under the hood

Learned Weights and Parameters

The adjustable numbers learned during training—and the count used to describe model size.

During training, the system repeatedly adjusts large collections of numbers called learned weights so its predictions improve. These include the embedding table and other large grids of numbers used throughout the model. Training mainly captures statistical patterns, although models can sometimes memorize details or passages from their data.

A parameter is one adjustable value in the model. Parameter count is therefore a rough measure of model size. More parameters can provide more capacity, but architecture, data, training, tools, and task fit also matter. A smaller model can outperform a larger one on a particular job.

Each slider is one weight — training just nudges these until the output sounds right.

What parameter count means

Every dot is one adjustable number — real models have billions of them, not a few dozen.
Why it matters: These terms explain why model files can be huge and why running larger models generally requires more memory and computation.
Think of it like: a mixing board with billions of adjustable settings. Parameter count tells you how many controls exist; training determines where they end up.
Common misconception: A bigger parameter count is not a universal quality score.
Back to contents →
Under the hood

Attention Scores and Transformers

How a language model connects each token with the earlier context that matters.

A transformer is the main design behind today's large language models. It processes text through a series of layers that gradually build a richer understanding of the prompt.

Within those layers, attention scores influence how strongly one token draws on other available tokens. In a text-generating model, a position cannot use future tokens that have not been generated yet. The model performs several attention calculations in parallel so it can notice different kinds of relationships. When producing an answer, it still generates one token at a time.

Working out what "it" refers to — "trophy" gets a much higher attention score than "suitcase."

The transformer architecture

Each token can draw on the earlier tokens available to it, with stronger lines showing greater influence.
Why it matters: Attention helps the model connect pronouns, topics, instructions, and other relationships across the available context.
Think of it like: several teams of editors, each marking different relationships in the same material before passing their notes to the next team.
Common misconception: An attention score is calculated for the current input. It is different from the model's learned settings, called weights, which are explained in the preceding section.
Back to contents →