Models, Harnesses, and Tools

Modern Plain Text Social Science
Week 12

Kieran Healy

Duke University

November 2026

Models, Harnesses, and Tools

Load the packages, as always

library(here)      # manage file paths
library(socviz)    # data and some useful functions
library(tidyverse) # your friend and mine
library(btw)       # describe R things to LLMs

The plan

  1. What a Large Language Model actually is
  2. The difference between the model and its harness
  3. Tools, and why they make the output better
  4. Subscription vs API access
  5. Bringing R to the LLM with btw
  6. Working with agents safely

The Model

What an LLM is

  • A very large function: text in, text out
  • Strictly, tokens in, a probability distribution over the next token out. Sample one, append it, repeat.
  • Trained on an enormous amount of text (including a great deal of R code), then tuned to follow instructions and hold a conversation.
  • Its weights are fixed after training. Its knowledge stops at a training cutoff.

What an LLM is not

On its own, the model cannot:

  • Run code
  • Look anything up
  • Read your files
  • Know today’s date
  • Remember your last conversation
  • Learn anything, in the sense we usually use that word.

When a chat app seems to do these things, the work is being done outside the model.

The model as a web API

You send a JSON request:

POST https://api.anthropic.com/v1/messages

{
  "model": "claude-...",
  "max_tokens": 1024,
  "system": "You are a helpful assistant.",
  "messages": [
    {"role": "user", "content": "What's the median of 3, 5, 10, and 12?"}
  ]
}

The model as a web API

And you get JSON back:

{
  "role": "assistant",
  "content": [
    {"type": "text", "text": "The median is 7.5."}
  ],
  "stop_reason": "end_turn",
  "usage": {"input_tokens": 31, "output_tokens": 9}
}

Note the usage field. Tokens are the unit of both cost and capacity.

It is stateless

  • The model keeps no record of the conversation between requests.
  • So every request has to carry the whole conversation so far.
  • The “Pachinko Machine” picture.
  • The “memory” in a chat window is just the prior turns being re-sent each time.
  • Consequences:
    • Long conversations cost more per message, and get slower.
    • There is a hard ceiling: the context window.
    • Quality degrades as the context fills up with irrelevant material.

Some vocabulary

  • Turn: one message, from either the user or the assistant.
  • Context: everything sent to the model in a single request: system prompt, prior turns, tool definitions, tool results.
  • Tool: a function the model can ask to have run.
  • Round: everything that happens between your request and the final answer. Possibly many turns.

The model alone is confidently wrong about some things

Ask a bare model, with no tools, to multiply two eight-digit numbers.

It will produce a number. The number will have the right number of digits, and the right first few digits. It will look entirely plausible.

x <- 20598162
y <- 83106206

format(x * y, big.mark = ",", scientific = FALSE)
[1] "1,711,835,094,393,372"

One bare model answered 1,711,067,239,026,172, which is close but wrong.

Why the model gets this wrong

  • The model produces plausible continuations. For arithmetic on numbers it has never seen, plausible is not the same as correct.
  • The same thing happens with R code:
    • Arguments that don’t exist, or that used to exist
    • Superseded idioms: %>%, gather()/spread(), aes_string(), size for line widths
    • Functions from packages newer than the training cutoff simply aren’t known
  • A better model helps less here than giving the model a way to check its answer.

The Harness

The model and the harness

  • “Claude” the app is not the same thing as “Claude” the model.
  • A harness is the program wrapped around the model. It:
    • Manages the conversation and re-sends it
    • Writes a system prompt
    • Reads context files from disk
    • Tells the model what tools exist
    • Runs the tools the model asks for
    • Enforces permissions
    • Gives you an interface

Harnesses you may have met

  • claude.ai and the Claude Desktop app
  • Claude Code, in the terminal or inside an editor like VS Code or Positron
  • Other coding agents: Codex, Gemini CLI, Positron Assistant, GitHub Copilot
  • ellmer, an R package: your R session becomes the harness

The same underlying model behaves very differently in each, because the harness decides what the model sees and what it can do.

Where things live

  You  <──────>  Harness  <──── JSON over HTTPS ────>  Model
                    ▲
                    │
                    ▼
          Tools: files, R, shell, web
  • The harness holds the system prompt, context files, conversation history, and tool definitions. It runs the tools.
  • It may run on your machine (Claude Code, ellmer) or on a server (claude.ai).
  • The model only ever receives text, and only ever returns text. (A consequence of this: there is no more “secret”, “underlying”, or “lower-level” language that “programmers” or Anthropic, or whomever, speak to the model in. It’s all just natural-language text top to bottom.)

Who does what

Component Provided by
Predicting the next token Model
System prompt Harness
Re-sending earlier turns Harness
Reading CLAUDE.md / AGENTS.md Harness
Describing the available tools Harness
Actually running a tool Harness
Asking you for permission Harness

Most of the differences you notice day to day are harness differences.

Context files are just text

This course’s repository has an AGENTS.md file. Part of it:

# Constraints

- Some datasets used may be very large. Do NOT attempt to ingest them in
  order to inspect them. Use the available local tools (R and its packages;
  the package APIs that allow the data to be queried, your plugins and
  skills, etc). You may extract very small samples (<1000 rows) from data
  files if it's necessary.

- The user will handle git and other version control matters.

The harness pastes this into the context at the start of every session, which is how “project memory” works.

Tools

What a tool is

  • A function that the harness runs on the model’s behalf.
  • The model sees only three things about it:
    1. A name
    2. A description, in plain English
    3. A schema for its arguments
  • The model emits a structured request: “call read_file with path = 'R/clean.R'”.
  • The harness runs it and sends the result back as text, in the next request.

Read tools and write tools

  • Read tools observe the world: list files, read a file, read a help page, describe a data frame, check git status.
  • Write tools change the world: write or edit a file, run a shell command, make a commit.

The loop

sequenceDiagram
  participant U as You
  participant H as Harness
  participant M as Model
  U->>H: "Plot donors by year for each country"
  H->>M: prompt + tool definitions
  M->>H: tool request: describe organdata
  H->>M: tool result: columns, types, ranges
  M->>H: tool request: run this ggplot code
  H->>M: tool result: error, no column 'donor'
  M->>H: tool request: run the corrected code
  H->>M: tool result: success
  M->>H: final answer
  H->>U: "Here's the plot ..."

Agents

An agent is an LLM, in a harness, that calls tools repeatedly in a loop.

The loop: read, act, read again to check what happened, decide what to do next.

A tool is just a function

Here’s a calculator. It evaluates an expression in an environment that contains only arithmetic.

calculator <- function(code) {
  math_env <- new.env(parent = emptyenv())
  ops <- c("+", "-", "*", "/", "^", "(", "sqrt", "exp", "log")
  walk(ops, \(op) assign(op, get(op, envir = baseenv()), envir = math_env))
  eval(parse(text = code), envir = math_env)
}

calculator("20598162 * 83106206") |>
  format(big.mark = ",", scientific = FALSE)
[1] "1,711,835,094,393,372"

A tool is just a function

And because of how it’s built, it can’t do anything else:

calculator("read.csv('secrets.csv')")
Error in `read.csv()`:
! could not find function "read.csv"

Give the model this tool, describe it as “use this for any arithmetic, since you can’t reliably compute it yourself”, and it gets the multiplication right.

What the model sees of a tool

Here is one of the tools in the btw package, as the model receives it:

help_tool <- btw_tools("btw_tool_docs_help_page")[[1]]

help_tool@description
[1] "Get help page from package."
help_tool@arguments@properties |>
  map_chr(\(p) p@description) |>
  str_trunc(70)
                                                            package_name 
"The exact name of the package, e.g. 'shiny'. Can be an empty string..." 
                                                                   topic 
"The topic_id or alias of the help page, e.g. 'withProgress' or 'inc..." 
                                                                 _intent 
"The intent of the tool call that describes why you called this tool..."

That text is all the model knows about this tool, so the descriptions have to be written carefully.

Why tools improve the output

  1. Ground truth instead of plausible guesses. A calculator. The help page for the version of dplyr you actually have installed.
  2. Current information. Today’s date. Package release notes. The web.
  3. Verification. Run the code. Run the tests. Render the document. Look at the error. Try again.
  4. Acting, not just suggesting. Edit the file directly, instead of handing you a snippet to paste.

Of the four, verification matters most: a model that can check its own work is much more reliable than one that can’t.

Tools also manage the output

  • An edit tool that replaces one exact span of text fails loudly if the text isn’t found. That’s safer than having the model rewrite a whole file from memory.
  • Structured output: constrain the response to a schema, so you get a data frame back, not prose.
  • Deterministic steps belong to deterministic tools. Don’t ask a model to format code. Run air format .
  • A context-file instruction (“always run air”) is a request. A hook that runs air after every edit is a guarantee.

A coding agent is a handful of tools

  1. Read a file
  2. Write a file
  3. Edit part of a file
  4. List files
  5. Search across files
  6. Run a shell command

Only 1, 2, and 6 are strictly essential. Claude Code’s built-in tools are essentially these, plus web search and fetch, and the ability to start sub-agents.

Subscription vs API

Two ways to pay for the same models

Subscription API
You pay A flat monthly fee Per token, input and output
Limits Usage caps that reset over time windows Rate limits, plus spend limits you set
Credentials Log in with your account An API key, e.g. ANTHROPIC_API_KEY
Harness Anthropic’s: claude.ai, Desktop, Claude Code Anything: Claude Code, ellmer, your own
Good for Interactive work Programs, pipelines, batch jobs, apps

API keys for your own code

  • A subscription login is for use with the provider’s own apps.
  • If your code calls the model directly, it needs an API key. That includes ellmer::chat_anthropic() and btw::btw_client().
  • API keys are secrets.
    • Keep them in ~/.Renviron, never in a script.
    • Never commit them to git.
    • Set a spend limit in the console.

Two directions

Bring the LLM to R

  • ellmer in your R session. R is the harness.
  • Needs an API key.
  • Good for using an LLM as part of an analysis, e.g. classifying 10,000 open-ended survey responses.

Bring R to the LLM

  • An existing harness, like Claude Code, calls R through tools.
  • Works with a subscription.
  • Good for interactive work: writing, checking, and fixing code with your data and your packages.

For reference: R as the harness

This needs an API key, so we won’t run it today. The whole harness is about a dozen lines.

library(ellmer)

chat <- chat_anthropic()   # needs ANTHROPIC_API_KEY

chat$register_tool(tool(
  calculator,
  description = "Evaluate a mathematical expression. Use this for any
    arithmetic, since you can't reliably compute it yourself.",
  arguments = list(code = type_string("An expression, e.g. '1 + 2 * 3'."))
))

chat$chat("What's 20598162 * 83106206?")
#> ◯ [tool call] calculator(code = "20598162 * 83106206")
#> ● #> 1711835094393372
#> 20598162 × 83106206 = 1,711,835,094,393,372.

Bringing R to the LLM

The Model Context Protocol (MCP)

  • A standard way for a harness to discover and call tools provided by some other program.
  • The harness is the MCP client. The program providing tools is an MCP server.
  • Write the server once and it works with any client: Claude Code, Claude Desktop, Positron, VS Code.
  • If R can run an MCP server, any of these harnesses can use R as a tool.

The btw package

“By the way, here’s some context.” A toolkit for describing R things to LLMs.

Three ways to use it:

  1. btw() at the console: copies a description to the clipboard, for pasting into any chat.
  2. btw_mcp_server(): serves R tools to Claude Code, Claude Desktop, etc.
  3. btw_client(): an ellmer chat with btw tools. Needs an API key.

Modes 1 and 2 work with a subscription.

Mode 1: btw() at the console

organ_small <- organdata |>
  select(country, year, donors, consent_law)

btw(organ_small, clipboard = FALSE)
## Context

organ_small
```json
{"n_cols":4,"n_rows":238,"groups":[],"class":["tbl_df","tbl","data.frame"],"columns":{"year":{"variable":"year","type":"Date","min":"1991-01-01","max":"2002-01-01","median":"1996-07-02","n_unique":12},"country":{"variable":"country","type":"character","min":5,"max":14,"empty":0,"n_unique":17,"values":["Australia","Austria","Belgium","Canada","Denmark","Finland","France","Germany","Ireland","Italy"]},"consent_law":{"variable":"consent_law","type":"character","min":8,"max":8,"empty":0,"n_unique":2,"values":["Informed","Presumed"]},"donors":{"variable":"donors","type":"numeric","mean":16.4848,"sd":5.1108,"p0":5.2,"p25":13,"p50":15.1,"p75":19.6,"p100":33.9}}}
```

Interactively, btw(organ_small) puts this on your clipboard, ready to paste into claude.ai.

Mode 1: btw() at the console

You can mix objects, documentation, and your question:

btw(
  organ_small,
  dplyr::across,
  vignette("colwise", "dplyr"),
  "Using across(), get the mean and sd of every numeric column by consent_law."
)
#> ✔ btw copied to the clipboard!
  • The model gets your actual column names and types, not guesses.
  • It gets the help page from your installed version of dplyr.
  • This is crude, but it works with any chat interface, and it’s much better than giving the model no context.

Mode 2: btw as an MCP server

Tell Claude Code about it, once, from the terminal:

claude mcp add -s user r-btw -- Rscript -e \
  "btw::btw_mcp_server(list('docs', 'env', 'sessioninfo', 'cran'))"

Check that it’s connected:

claude mcp list
#> r-btw: Rscript -e btw::btw_mcp_server(...) - ✔ Connected

Now, in every Claude Code session, the model can call R’s help system, inspect data frames, check installed packages, and search CRAN.

Choosing tool groups

btw_tools() |>
  map(\(t) tibble(group = t@annotations$btw_group, tool = t@name)) |>
  list_rbind() |>
  summarize(tools = n(), .by = group)
# A tibble: 12 × 2
   group       tools
   <chr>       <int>
 1 agent           1
 2 cran            3
 3 docs            5
 4 env             2
 5 files           6
 6 git             7
 7 github          1
 8 ide             1
 9 pkg             5
10 sessioninfo     3
11 skills          1
12 web             1
  • Claude Code already has its own file, search, and shell tools. Don’t duplicate them.
  • docs, env, sessioninfo, and cran are a good start. Add pkg if you write packages.
  • btw_tool_run_r is opt-in. Name it explicitly to let the model run R code.

Docs in the loop

Ask a model about purrr::in_parallel(), added in mid-2025. Without tools, it may not know it exists, or may confidently invent arguments. With btw, it calls a tool instead:

help_lines <- btw_tool_docs_help_page("in_parallel", "purrr")@value |>
  str_split_1("\n")

help_lines[str_which(help_lines, "#### Usage"):length(help_lines)] |>
  head(16) |>
  writeLines()
#### Usage

``` R
in_parallel(.f, ...)
```

#### Arguments

##### `.f`

A fresh formula or function. "Fresh" here means that they should be
declared in the call to `in_parallel()`.

##### `...`

Named arguments to declare in the environment of the function.

Connecting to your live R session

  • By default, the MCP server runs in its own fresh R process. It can’t see the objects in your console.
  • To share your session, run this in your interactive R console (RStudio or Positron):
btw::btw_mcp_session()
  • Now the env tools describe your data frames, and run_r runs in your session.
  • The model calls list_r_sessions and select_r_session to find it.
  • If Claude Code was already running, reconnect with /mcp.

An interactive workflow

In your R console:

library(tidyverse)
library(socviz)

organ_small <- organdata |>
  select(country, year, donors, consent_law)

btw::btw_mcp_session()

In Claude Code:

Using `organ_small` in my R session, make a faceted line plot of donors
by year for each country, grouped by consent_law. Use current ggplot2
conventions. Run the code to check that it works before showing it to me.

The tool calls, step by step

Step Tool call Why it matters
1 select_r_session Attaches to your console
2 btw_tool_env_describe_data_frame Real column names, types, ranges
3 btw_tool_docs_help_page (ggplot2) Current arguments, e.g. linewidth not size
4 btw_tool_run_r Runs the code; sees warnings and errors
5 btw_tool_run_r (again) Fixes the problem and re-checks
6 Final answer Code that has actually been run

The plot appears in your session’s plot pane, because the code ran in your session.

Why this beats copy-and-paste

  • Documentation from the versions you have installed.
  • Data: actual names and types, without uploading the data itself.
  • Verification: code is run before you see it.
  • Iteration: you keep working in your console; the model works alongside it on the same objects.
  • Package work: with the pkg group, the model can document(), test(), and check() in the loop.

The btw command-line tool

Some harnesses can’t use MCP, or you want to keep tool definitions out of the context. btw also ships a CLI.

btw::install_btw_cli()
btw docs topics purrr
btw docs help dplyr::across
btw pkg test

Any agent with a shell tool can call it. The installer offers a skill telling the agent the CLI exists.

Context files and skills

  • Context files: CLAUDE.md or AGENTS.md at the root of a project. Standing instructions: style rules, project layout, things to avoid.
  • Skills: folders of instructions (and sometimes scripts) that the harness loads only when relevant. Cheap until needed.
  • R packages can ship skills. btw_skill_install_package() installs them; btw_skill_install_project() does it for all of a project’s dependencies.
  • These are plain-text files, kept in the project and versioned with git like everything else in it.

Working safely

Agents act

  • Write tools change things. Acting on your behalf is what makes an agent useful, and it also means its mistakes have real consequences.
  • Harnesses offer permission modes: ask before every action; allow edits but ask about commands; plan only; or decide automatically.
  • Sandboxes restrict which files and which network hosts a command can touch.

“Auto” doesn’t mean “careful”

Hadley Wickham asked two coding agents in auto mode to install a tool via curl ... | sh, i.e. run a script from the internet without reading it.

  • Both recognized this was risky.
  • Both ran it anyway, because the user had asked.
  • One disabled its own sandbox when the sandbox got in the way.

The model’s judgment is not a security boundary.

Practical safeguards

  • git is your safety net. Commit before you let an agent loose. Read the diff afterwards.
  • Keep secrets out of reach. .Renviron for keys, not scripts.
  • Large or sensitive data: tell the agent to query it (with duckdb, arrow, dbplyr), not read it.
  • Start with read-only tools. Add write tools as you gain confidence.
  • Prefer narrow tools with simple inputs (like the calculator) to “run anything”.

Implications for your work

  • The model’s output is a draft. Your judgment is the check.
  • Verification matters more now, not less: tests, rendering, targets pipelines.
  • An LLM’s responses are not reproducible, but the code it writes is reproducible once you’ve read and committed it.
  • The things this course has covered (plain text, the shell, scripts, Quarto, git, build systems, packages) are what make a project legible to an agent.

Reading