feat: batched export, smart rate-limit backoff, filter chips, and dialog UX improvements (#348)
- feat(export): batch export in 100-conv waves and date-range filter
Closes #347 (feature request on upstream pionxzh/chatgpt-exporter).
Changes:
- src/constants.ts: add EXPORT_OPERATION_BATCH = 100 (fixed, non-tunable)
- src/utils/download.ts: add PartInfo type, part-suffix support to
buildZipFileName, and new buildJsonBatchFileName for official JSON
- src/exporter/{html,json,markdown}.ts: accept optional partIndex /
totalParts params; each exportAllTo* uses PartInfo to suffix its
output filename when multiple batches are produced
- src/ui/ExportDialog.tsx:
- Remove hardcoded EXPORT_LIMIT = 100 selection cap; users may now
select all loaded conversations (up to exportAllLimit from settings)
- Add DateFilter component with from/to date inputs that filter the
conversation list by create_time (inclusive, end-of-day for dateTo)
- Wave-based export pipeline: splits selected conversations into
chunks of EXPORT_OPERATION_BATCH (100), fetches each chunk via
RequestQueue, downloads the completed batch, pauses 400 ms, then
continues with the next chunk — keeping memory bounded and API
pressure low
- Progress bar now shows global count across all batches plus
"Batch X/N" indicator when multiple batches are active
- Local-file source also batched in 100-conversation waves with the
same 400 ms inter-batch delay
- Shift-click range selection and Select All no longer impose a
hard cap (handled by the batch pipeline instead)
- src/locales/en.json: add Date Filter Label/Hint, Selected count,
Export batch info, Exporting batch strings; update Export All Limit
Description to mention 100-conversation waves
Made-with: Cursor
- fix(export): correct date filter for ISO string timestamps + add update_time field
The list endpoint returns create_time/update_time as ISO 8601 strings
(e.g. "2025-07-10T14:53:58.103234Z"), not Unix numbers. The previous
filter compared a string to a number, always producing NaN comparisons
and filtering out every conversation.
- Add toMs() helper that handles both ISO strings and legacy Unix numbers
- Fix filter comparisons to use toMs() and compare in milliseconds
- Update ApiConversationItem type: create_time is number|string,
update_time is optional number|string
- Add filterField state ('create_time' | 'update_time') with a
Created/Updated selector in the DateFilter UI so users can filter
by either timestamp
- Add i18n strings for the new selector labels
Made-with: Cursor
feat: redesign date filter layout with presets, auto-load conversations, batch label on export
Date filter redesigned to two-row layout: field selector + quick preset
buttons (7d / 30d / 90d / This year / Clear) on row 1; From–To date
inputs on row 2, eliminating the wrapping issue
Auto-load all conversations when dialog opens (default project null
instead of undefined) so search box, Last 100 button, and counter are
immediately visible without needing to manually pick a project
Export button shows batch count when selection > 100, e.g. "Export · 3 files"
with a tooltip explaining "3 separate downloads, 100 conversations each"
Progress display shows "Batch X/Y · completed/total" for multi-batch exports
Remove redundant "Export from API" static text (project selector implies it)
Add locale strings: Date Preset 7d/30d/90d/Year, Clear filter, Batch progress
Made-with: Cursor
- fix: show batch download count inline below export button
Made-with: Cursor
feat: filter chips, date column, All conversations, starred indicator
Filter chips via # trigger: type # in search box to open a popover
with available filters; click to add as a removable chip:
★ Starred only | 💾 Saved only (skip temp) | ⏱ Long conversation
(active ≥ N days, editable inline) | 🤖 GPT/custom AI | 💬 Regular chats
Date column: each conversation row now shows a compact relative date
(Today / Yesterday / Xd ago / Mon DD / MMM YYYY) on the right edge
Starred indicator: ★ shown inline for starred conversations
Project dropdown: new "📂 All conversations" option fetches the main
list plus every project's conversations, deduplicated; uses new
fetchAllConversationsAll() API function
ApiConversationItem: add is_starred, is_temporary_chat, gizmo_id fields
from the list endpoint payload
CSS: SelectItem flex layout, new SelectChip / SelectChips /
SelectFilterPopover / SelectItemMeta classes
Made-with: Cursor
feat: add Load More button to fetch conversations beyond the initial limit
Export fetchConversationsPage from api.ts for single-page incremental loading
Add onHasMore callback to fetchAllConversations so callers know when the
fetch was stopped by the user-configured limit (vs the API having no more data)
DialogContent tracks hasMore / totalAvailable / loadingMore state
A "Load N more" button appears below the conversation list when more pages
exist; shows "N remaining" count once the API total is known
Resets cleanly on project change
Made-with: Cursor
feat: comprehensive filter chip system with chat class, origin, status, and mode logic
Add conversation_origin, pinned_time, is_archived, is_do_not_remember to ApiConversationItem
Replace simple chip flags with typed FilterChipDef union:
chat_class (Regular/GPT/Project), origin (dynamic from data),
status (starred/temporary/pinned), duration_gte, recency_lte
Each chip has an include/exclude mode toggle (click badge to flip)
AND/OR logic toggle appears in chip bar when 2+ chips are active
Text search supports * and ? wildcards alongside chips
Picker grouped by category with dynamic Origin values from loaded conversations
Project classification cross-references fetched project IDs against gizmo_id
Remove ProjectSelect dropdown: always auto-load all conversations on open
Chat class chip: Regular (no gizmo), GPT (store gizmo), Project (matched gizmo)
Made-with: Cursor
refactor(filters): redesign chips as multi-select with two-stage picker
Chat class, Origin, Project, Status are now single chips that open a
sub-view where users tick multiple values (Regular + GPT, etc.)
New Project chip lets users pick specific project(s) by name
Date filter moved from top-of-dialog into the chip bar as a full-width
date-range chip (Created/Updated selector + date inputs + presets)
Picker uses two-stage UX: root list → sub-view with checkboxes + Apply
Clicking a chip's value label reopens the picker for in-place editing
AND/OR logic toggle only appears for 2+ non-date chips
Remove standalone DateFilter component and related state from DialogContent
Made-with: Cursor
- fix(filters): guard gizmo_id null check for project chip
Made-with: Cursor
fix: prevent export dialog close mid-run and add resume-from-offset control
Block ESC, outside-click, and X button while an export is in progress;
module-level exportingRef bridges DialogContent to ExportDialog without
lifting state.
Stale-fetch prevention: fetchGenRef generation counter discards callbacks
from a previous dialog mount after remount, eliminating the console
errors seen when reopening the dialog mid-run.
Cleanup on unmount: clear all three RequestQueues when DialogContent
unmounts so orphaned background work stops immediately.
Add "→ 100 from #N" resume control in the toolbar: a number input sets
the starting offset and the arrow button selects the next 100 convs
from that position, letting users continue a partial export (e.g. set
offset to 200 to resume after 2 finished batches).
Made-with: Cursor
- perf: use Web Worker for sleep() to bypass background-tab timer throttle
Chrome throttles setTimeout in non-focused tabs to >=1 s minimum, which
causes the export queue (200 ms between requests * 100 convs = 20 s/batch)
to stall entirely when the user switches away from the tab.
Fix: run all timers inside a single persistent Web Worker. Workers are
exempt from background-tab throttling, so exports run at full speed
regardless of tab focus. A graceful fallback to regular setTimeout is
retained for environments where blob-URL Worker creation fails (CSP etc.)
so the change is entirely non-breaking.
Made-with: Cursor
- fix: proper 429 backoff — wait Retry-After secs, cap retries, skip when exhausted
Previously the queue retried forever with a max backoff of 1600 ms, causing
the 429 storm seen in the console (the same request hammering the API
continuously with no meaningful pause).
Changes:
- api.ts: export RateLimitError with retryAfterMs read from Retry-After
header (defaults to 30 s when header is absent or unparseable)
- queue.ts: on RateLimitError, sleep max(retryAfterMs, 30 s) before
retrying; give up after 8 consecutive 429s and skip the request; on
regular errors give up after 5 retries and skip; add rate_limited status
so the UI can show what is happening
- ExportDialog.tsx: show "Rate limited — waiting Xs…" with an amber
progress bar when status is rate_limited; clear progress bar to blue
once the pause ends
Made-with: Cursor
- fix: global queue pause on 429 instead of per-item backoff
The previous design retried each item individually with a 30s wait, so if
all 100 conversations in a batch were rate-limited the theoretical worst
case was 100 x 8 x 30s = 24,000s.
New design:
- On the first 429 the entire queue is frozen (pauseUntil timestamp).
Every subsequent process() call checks pauseUntil and waits out the
remaining time before making any new request — one shared wait for all
queued items, not N separate waits.
- The pause length is max(Retry-After header, 60s * pauseCount) so it
backs off exponentially if the first pause was not long enough:
pause 1 = 60s, pause 2 = 120s, pause 3 = 180s, etc.
- After MAX_GLOBAL_PAUSES (5) the queue aborts rather than looping forever
(total max wait ~15 min before giving up).
- The failed item is always returned to the front of the queue to be
retried after the pause clears rather than being counted as a skip.
- Removed per-item rateRetries counter (no longer needed).
Made-with: Cursor
feat: add Test API button and rate-limit header logging
probeApi(): make one minimal request (1 conversation) to check whether
the API is currently accepting traffic. Returns ok/rate_limited status
plus any X-RateLimit-* or Retry-After headers in the response.
fetchApi(): logs all known rate-limit response headers to console on
every call (success or failure) so we can discover which headers
ChatGPT actually sends.
ExportDialog: "Test API" button next to the upload icon. Shows
"Testing... / API ready / Rate limited · wait Xs / Error" inline.
Hovering the button after a successful test shows any rate-limit headers
as a tooltip (useful for debugging).
Made-with: Cursor
- fix: add Cancel button during export; replace useless dimmed X
Previously the X button was grayed out and non-functional while an export
was running, with no way to stop it short of refreshing the page.
Changes:
- Cancel button (red, top-right) appears while processing. Clicking it
sets cancelledRef, calls queue.stop() on all three queues, and lets the
in-flight request finish before cleanly stopping the loop.
- The on('done') handler checks cancelledRef: if set, it skips the
download and the next-batch kickoff and just resets processing state.
- cancelledRef is cleared at the start of each new export so back-to-back
exports work correctly.
- The dimmed X is kept (slightly more transparent) as a visual reminder
that the dialog cannot be closed mid-export; its tooltip now says
"Click Cancel to stop the export".
Made-with: Cursor
- feat: sortable column headers, both date columns, unambiguous date format
Problems fixed:
- Date column only showed create_time with no label — confusing when the
user had filtered or sorted by update_time.
- Dates < 1 year omitted the year ("Jun 17" vs "Jun 17, 2025"), making it
impossible to tell which year a conversation belonged to.
- No way to sort the list — order was whatever the API returned.
Changes:
- formatConvDate: always includes the year (removes the < 365-day branch).
Still shows "Today" / "Yesterday" for very recent items.
- ConversationSelect: two date columns (Created, Updated). The active-sort
column is visually highlighted (bold, blue) so the user always knows
what order the list is in.
- Sortable header row (SelectListHeader): click Title / Created / Updated
to sort; click again to toggle asc/desc. Arrow indicator (↑/↓/↕) shows
current state. Defaults to Created ↓ (newest first) matching the
original API order.
- Sorting applied in the filtered memo after chip/text filtering.
Made-with: Cursor
Co-authored-by: Pionxzh [email protected]
