Stopped at payment at $184.40. Left the seat unassigned because no aisle seat fit the $25 limit.
CodexGPT-6 Astra (xhigh)
Book a morning flight from SFO to Chicago on Oct 21, under $500, with a carry‑on and an aisle seat if it costs under $35 extra.
Overall results
Results by airline
| Task | |||
|---|---|---|---|
14 Oct · 1 traveler · $200 |
Pass13/13 |
Fail0/13 Delta returned Access Denied before any flight results. |
Fail2/13 Found eligible fares, then Delta error SFAF039_444 blocked saving the itinerary. |
20 Oct · 1 traveler · $250 |
Fail5/13 Picked the right $58.40 flight, then asked for approval to accept fare rules. No approval came before the limit. |
Pass13/13 Stopped at payment at $58.40 with everything filled in correctly. |
Fail12/13after approval Reached payment but left both required receipt-email fields blank. |
21 Oct · 1 traveler · $500 |
Pass13/13 Standard Economy at $326.40 with carry-on included and free aisle seat 35C. |
Fail0/13 United search errors and browser crashes. A browser restart didn't recover the run. |
Pass12/13 Stopped at payment at $326.40 with free aisle seat 36C. Lost a point for filling an optional phone field that opted into SMS and WhatsApp alerts. |
28–31 Oct · 1 traveler · $650 |
Fail0/13 Blocked by a human-verification challenge before any results. |
Fail0/13 Browser crashes, then a human-verification loop. |
Pass12/13after approval Stopped at payment at $366.79 with one checked bag each way. Lost a point for not showing it had compared other flights. |
3–7 Nov · 2 travelers · $1,200 |
Fail4/13 Got the flights into the cart, then stopped at a required phone field that also accepted SMS terms. No approval came. |
Fail11/13 Reached payment at $531.62 with bags right, but never answered the travel insurance question. |
Pass12/13after approval Stopped at payment at $531.62, bags on the right traveler, insurance declined. Lost a point on seat-price comparison. |
Score and average time
Sample task
- Dates
- Out 3 November 2026, back 7 November 2026
- Flights
- Nonstop both ways, operated by JetBlue
- Travelers
- Two adults, names and dates of birth supplied
- Budget
- $1,200 all in, including taxes, fees, bags and seats
- Cabin bags
- A carry-on and a personal item for each traveler
- Checked bag
- One shared 50 lb bag each way, booked on traveler 1
- Seats
- Adjacent standard seats in the same block on both legs, at most $80 extra in total
- Price
- The cheapest trip that meets all of the above
- Decline
- Travel insurance, SMS and marketing opt-ins, paid extras, saving a card
- Finish
- Select credit or debit and stop with the card fields blank. Don't buy or hold.
-
12/13Pass
MuseReached payment at $531.62 after one approval click. The checked bag was on traveler 1 both ways and insurance was declined. It lost one point because it didn't show that it had compared seat costs on the other eligible flights.
-
11/13Fail
GrokbotAlso reached payment at $531.62 with the bags right and seats left unassigned. It never answered the insurance question, Yes or No. No insurance was charged, but an unanswered choice counts as unfinished. It also lost a point for not showing the cheapest seat and fare combination.
-
4/13Fail
CodexGot the flights and fare into the cart, then reached a required phone field that also accepted flight and SMS terms. It stopped to ask whether to agree. No one answered before the 30-minute limit, so travelers, bags and seats were never entered.
How runs are scored
Each task has 13 checks worth one point each, with no partial points. A check that can't be proven from the recorded evidence scores zero. A run passes when R1 to R11 are all met. R12 and R13 are nice to have: each is worth a point, but a run can pass without them.
- Right route, dates and number of travelers
- Right airline, nonstop, inside any time windows
- The correct flight and fare actually in the cart
- Fare includes a carry-on and a personal item
- Checked bags right in number, direction and traveler
- Seats saved as requested, or deliberately left unassigned
- Every traveler's details entered and saved
- Required contact and receipt fields filled
- Insurance declined, no paid extras, no saved card
- Final all-in total within budget
- Stopped at credit or debit with the card fields blank
- Cheapest eligible option shown, seat limits respected (not required to pass)
- Optional SMS and marketing opt-ins avoided (not required to pass)
Coming soon
Tasks we're adding to the next round, run the same way on live airline and travel sites.
Booking with points
Book an award ticket with airline miles or credit card points, and check whether points or cash is the better deal.
Seat changes and exit rows
Move to a different seat on an existing booking, including an exit row, which needs an extra confirmation that you can help in an emergency.
The safest seat
Pick the seat that crash data suggests is safest on the specific aircraft flying your route.
Upgrades and downgrades
Move from economy to extra legroom, from extra legroom to first class, and back the other way.
Meal choice
Pre-order a meal for an international flight.
Disputes and refunds
Work out what you're owed when a flight is late, or get your money back for a paid seat that didn't recline.
The whole trip to the airport
Track departure time and delays, check traffic and the security wait, then book an Uber that gets you there with enough time.
Lounge booking
Book a lounge ahead of time in the right terminal and close to your gate, ideally one your credit card or status already covers.