AirlineBench v0.1

Can AI assistants book flights on real airline websites?

Results from 24 September 2026

Every run used the airline's public website in real time, with real fares, real seat maps and real checkout flows.

Rubric score across five tasks

Muse76.9%
Codex53.8%
Grokbot36.9%

Results by airline

Task Codex Grokbot Muse
LAX → LAS, one way
14 Oct · 1 traveler · $200
13/13Pass

Stopped at payment at $184.40. Left the seat unassigned because no aisle seat fit the $25 limit.
0/13Fail

Delta returned Access Denied before any flight results.
2/13Fail

Found eligible fares, then Delta error SFAF039_444 blocked saving the itinerary.
LAX → LAS, one way
20 Oct · 1 traveler · $250
5/13Fail

Picked the right $58.40 flight, then asked for approval to accept fare rules. No approval came before the limit.
13/13Pass

Stopped at payment at $58.40 with everything filled in correctly.
12/13Failafter approval

Reached payment but left both required receipt-email fields blank.
SFO → ORD, one way
21 Oct · 1 traveler · $500
13/13Pass

Standard Economy at $326.40 with carry-on included and free aisle seat 35C.
0/13Fail

United search errors and browser crashes. A browser restart didn't recover the run.
12/13Fail

Same $326.40 fare and free aisle seat 36C, but filled an optional phone field that opted into SMS and WhatsApp alerts.
DFW ⇄ MCO, round trip
28–31 Oct · 1 traveler · $650
0/13Fail

Blocked by a human-verification challenge before any results.
0/13Fail

Browser crashes, then a human-verification loop.
12/13Passafter approval

Stopped at payment at $366.79 with one checked bag each way. Lost a point for not showing it had compared other flights.
JFK ⇄ FLL, round trip
3–7 Nov · 2 travelers · $1,200
4/13Fail

Got the flights into the cart, then stopped at a required phone field that also accepted SMS terms. No approval came.
11/13Fail

Reached payment at $531.62 with bags right, but never answered the travel insurance question.
12/13Passafter approval

Stopped at payment at $531.62, bags on the right traveler, insurance declined. Lost a point on seat-price comparison.

Score and average time

Muse76.9% · avg 11.8 min Codex53.8% · avg 5.8 min Grokbot36.9% · avg 12.8 min

Each icon marks one assistant: its rubric score across all five tasks against its average time per task. Time runs from the start of a task to where the run stopped or handed off, and failed runs count too. A low average isn't always good: Codex's includes a 1.2-minute run that ended at a human-verification check. Muse's includes waiting for three approval clicks. Per-run cost and token counts weren't recorded, so this chart can't show cost.

Show all 15 runs as a table
AssistantAirlineScoreMinutesOutcome

One task in full

The JetBlue task had the most moving parts. Every assistant got the same brief and a set of fictional traveler details.

JFK ⇄ FLLNew York to Fort Lauderdale, round trip
Dates
Out 3 November 2026, back 7 November 2026
Flights
Nonstop both ways, operated by JetBlue
Travelers
Two adults, names and dates of birth supplied
Budget
$1,200 all in, including taxes, fees, bags and seats
Cabin bags
A carry-on and a personal item for each traveler
Checked bag
One shared 50 lb bag each way, booked on traveler 1
Seats
Adjacent standard seats in the same block on both legs, at most $80 extra in total
Price
The cheapest trip that meets all of the above
Decline
Travel insurance, SMS and marketing opt-ins, paid extras, saving a card
Finish
Select credit or debit and stop with the card fields blank. Don't buy or hold.

  • Muse

    12/13Pass

    Reached payment at $531.62 after one approval click. The checked bag was on traveler 1 both ways and insurance was declined. It lost one point because it didn't show that it had compared seat costs on the other eligible flights.

  • Grokbot

    11/13Fail

    Also reached payment at $531.62 with the bags right and seats left unassigned. It never answered the insurance question, Yes or No. No insurance was charged, but an unanswered choice counts as unfinished. It also lost a point for not showing the cheapest seat and fare combination.

  • Codex

    4/13Fail

    Got the flights and fare into the cart, then reached a required phone field that also accepted flight and SMS terms. It stopped to ask whether to agree. No one answered before the 30-minute limit, so travelers, bags and seats were never entered.

How runs are scored

Each task has 13 checks worth one point each, with no partial points. A check that can't be proven from the recorded evidence scores zero. A run passes when every check except R6 is met.

  1. Right route, dates and number of travelers
  2. Right airline, nonstop, inside any time windows
  3. The correct flight and fare actually in the cart
  4. Fare includes a carry-on and a personal item
  5. Checked bags right in number, direction and traveler
  6. Cheapest eligible option shown, seat limits respected (not needed to pass)
  7. Seats saved as requested, or deliberately left unassigned
  8. Every traveler's details entered and saved
  9. Required contact and receipt fields filled
  10. Optional SMS and marketing opt-ins avoided
  11. Insurance declined, no paid extras, no saved card
  12. Final all-in total within budget
  13. Stopped at credit or debit with the card fields blank