Teknium
fb9f3a4ef9
fix(skills): pull full ClawHub catalog into the skills index (200 → 20k+) ( #33748 )
...
* fix(skills): pull full ClawHub catalog into the skills index
The website was showing 200 ClawHub skills out of 20k+ because
`ClawHubSource.search("")` for empty queries went straight to a single
unpaginated request. ClawHub's API caps any single page at 200 items and
returns a `nextCursor`; we grabbed page 1 and stopped, so the cached
index served from hermes-agent.nousresearch.com had a silent 99%
truncation.
End users never hit clawhub.ai directly (the index is rebuilt twice
daily by .github/workflows/skills-index.yml and served as a static JSON
on the docs site), so the cap-and-cache architecture is correct — it
just wasn't being filled.
Changes:
- `ClawHubSource.search(query="")` now routes through the existing
`_load_catalog_index()` paginating walker instead of the unpaginated
listing fallback (non-empty queries still hit the fast catalog search).
- `_load_catalog_index()` max_pages 50 → 250 (50k-skill ceiling; live
catalog is ~20k as of May 2026, with headroom for growth).
- `build_skills_index.py`: per-source crawl limits split out — ClawHub
and LobeHub get 100k, others keep their effective caps.
- `EXPECTED_FLOORS["clawhub"]` 50 → 5000 so the next pagination
regression hard-fails the CI build instead of silently shipping a
degenerate index.
Test plan:
- New unit test `test_search_empty_query_paginates_full_catalog`
exercises the cursor-following path with three mocked pages (450
total items) and asserts all pages are walked.
- Existing 9 ClawHub tests + 127 broader skills_hub tests all pass.
- E2E against live ClawHub API: walker reached 9700+ skills across 49
pages before this commit landed, paginating well past the previous
50-page cap.
* fix(skills): raise ClawHub ceilings — live catalog is 50k, not 20k
E2E walk against live ClawHub API hit my initial 250-page cap at 49,698
skills with cursor=yes still pending. The catalog is roughly 2.5x larger
than the docstring estimate.
- max_pages 250 → 750 (150k ceiling, walks terminate on cursor=None
well before this in practice)
- SOURCE_LIMITS['clawhub'] 100k → 200k
- EXPECTED_FLOORS['clawhub'] 5000 → 20000
2026-05-28 01:42:19 -07:00
..
2026-02-26 03:20:08 +03:00
2026-05-22 03:33:01 -07:00
2026-04-29 20:05:32 -07:00
2026-03-23 07:43:12 -07:00
2026-05-08 14:55:40 -07:00
2026-05-21 16:40:04 +05:30
2026-05-23 02:59:13 -07:00
2026-05-04 04:45:03 -07:00
2026-05-12 15:14:49 -07:00
2026-05-12 15:14:49 -07:00
2026-04-24 03:46:46 -07:00
2026-04-17 16:05:04 -07:00
2026-04-24 03:46:46 -07:00
2026-05-13 22:07:02 -07:00
2026-04-08 13:44:58 -07:00
2026-04-16 04:19:34 -07:00
2026-05-09 13:35:39 -07:00
2026-04-20 00:32:09 -07:00
2026-03-28 17:25:04 -07:00
2026-05-10 07:37:55 -07:00
2026-04-10 13:05:44 -07:00
2026-05-17 02:29:41 -07:00
2026-04-26 09:57:58 -07:00
2026-05-06 03:41:21 -07:00
2026-05-23 20:30:29 -07:00
2026-05-22 19:46:18 -07:00
2026-05-07 05:38:05 -07:00
2026-04-30 20:33:33 -07:00
2026-05-27 23:15:41 -07:00
2026-04-11 02:58:48 -07:00
2026-05-09 17:53:13 -07:00
2026-05-14 07:59:12 -07:00
2026-02-27 03:29:26 -05:00
2026-05-13 22:54:21 -07:00
2026-05-17 02:29:41 -07:00
2026-05-08 14:27:40 -07:00
2026-05-08 14:27:40 -07:00
2026-05-27 00:43:32 -07:00
2026-05-21 17:38:19 -07:00
2026-05-21 17:38:19 -07:00
2026-05-22 01:19:01 -07:00
2026-03-27 04:03:00 -07:00
2026-04-10 03:44:43 -07:00
2026-05-08 14:55:40 -07:00
2026-05-25 05:07:49 -07:00
2026-02-26 13:55:54 +03:00
2026-05-25 18:20:45 -07:00
2026-05-24 00:38:17 -07:00
2026-05-12 16:31:46 -07:00
2026-03-02 04:34:21 -08:00
2026-05-07 06:41:42 -07:00
2026-04-24 04:58:32 -07:00
2026-03-26 14:50:26 -07:00
2026-05-17 11:40:05 -07:00
2026-05-21 16:40:04 +05:30
2026-05-24 18:05:33 -07:00
2026-04-14 21:20:37 -07:00
2026-05-07 06:17:49 -07:00
2026-05-25 12:24:58 +10:00
2026-04-21 06:14:25 -07:00
2026-04-17 19:04:11 -07:00
2026-05-05 04:54:17 -07:00
2026-05-24 17:47:24 -07:00
2026-04-17 19:26:40 -07:00
2026-05-25 05:50:29 -07:00
2026-04-23 15:11:52 -07:00
2026-04-21 16:41:26 -07:00
2026-05-09 11:12:03 -07:00
2026-04-10 03:01:46 -07:00
2026-04-10 21:14:32 -07:00
2026-04-20 00:58:16 -07:00
2026-04-08 17:23:15 -07:00
2026-05-25 01:55:59 -07:00
2026-04-13 05:15:05 -07:00
2026-03-14 11:27:02 -07:00
2026-05-25 15:18:45 -07:00
2026-05-27 00:43:32 -07:00
2026-05-04 05:08:51 -07:00
2026-05-17 02:29:41 -07:00
2026-05-21 16:40:04 +05:30
2026-05-15 01:33:13 -07:00
2026-04-23 03:01:18 -07:00
2026-04-21 13:35:31 -07:00
2026-04-28 22:16:08 -07:00
2026-04-14 02:18:38 -07:00
2026-05-18 21:01:14 -07:00
2026-05-28 00:19:31 -07:00
2026-05-14 08:03:40 -07:00
2026-05-25 15:18:45 -07:00
2026-05-17 23:28:45 -07:00
2026-04-19 11:27:50 -07:00
2026-05-27 00:43:32 -07:00
2026-05-04 15:31:47 -07:00
2026-05-15 16:25:31 -07:00
2026-05-24 06:54:16 -07:00
2026-04-23 05:15:37 -07:00
2026-04-09 16:24:53 -07:00
2026-05-28 00:19:31 -07:00
2026-05-28 00:19:31 -07:00
2026-05-17 02:29:41 -07:00
2026-04-16 12:36:49 -07:00
2026-05-17 02:29:41 -07:00
2026-04-21 05:19:03 -07:00
2026-05-28 00:55:55 -07:00
2026-04-29 20:05:32 -07:00
2026-05-07 06:33:57 -07:00
2026-05-07 07:14:16 -07:00
2026-05-16 13:06:56 -07:00
2026-04-19 16:31:07 -07:00
2026-04-19 16:31:07 -07:00
2026-04-16 21:57:10 -07:00
2026-04-16 21:57:10 -07:00
2026-05-07 05:35:33 -07:00
2026-05-25 05:37:30 -07:00
2026-05-15 14:00:01 -07:00
2026-04-16 21:57:10 -07:00
2026-05-07 07:08:04 -07:00
2026-05-17 13:33:26 -07:00
2026-05-15 14:00:01 -07:00
2026-04-16 21:57:10 -07:00
2026-04-04 10:18:57 -07:00
2026-05-07 06:32:45 -07:00
2026-05-24 04:44:59 -07:00
2026-05-07 07:39:50 -07:00
2026-04-14 10:18:05 -07:00
2026-05-07 07:03:21 -07:00
2026-05-25 14:52:24 -07:00
2026-05-08 09:27:26 -07:00
2026-05-08 09:27:26 -07:00
2026-04-23 15:14:11 -07:00
2026-04-12 06:18:05 +05:30
2026-05-27 00:43:32 -07:00
2026-04-10 03:01:46 -07:00
2026-05-27 02:22:08 -07:00
2026-04-05 12:46:07 -07:00
2026-04-22 14:45:50 -07:00
2026-05-25 15:18:45 -07:00
2026-05-20 01:46:40 -05:00
2026-05-25 06:13:36 -07:00
2026-05-23 20:30:29 -07:00
2026-04-13 16:32:04 -07:00
2026-05-13 17:32:22 -07:00
2026-04-23 15:11:52 -07:00
2026-05-18 10:37:35 -07:00
2026-03-17 02:02:57 -07:00
2026-05-24 18:05:33 -07:00
2026-05-18 22:19:50 -07:00
2026-05-25 05:34:31 -07:00
2026-05-17 23:28:45 -07:00
2026-04-26 11:55:02 -07:00
2026-04-20 13:24:15 -07:00
2026-05-17 02:29:41 -07:00
2026-04-10 03:44:43 -07:00
2026-04-01 04:19:19 -07:00
2026-05-17 02:29:41 -07:00
2026-05-08 14:55:40 -07:00
2026-04-01 04:19:19 -07:00
2026-05-07 06:13:37 -07:00
2026-03-04 05:30:43 -08:00
2026-03-02 02:00:09 -08:00
2026-05-23 17:47:26 -07:00
2026-05-25 01:51:27 -07:00
2026-05-19 14:17:38 -07:00
2026-05-28 01:42:19 -07:00
2026-05-27 13:39:58 -07:00
2026-05-27 13:39:58 -07:00
2026-05-27 00:43:32 -07:00
2026-04-29 21:56:47 -07:00
2026-04-24 07:06:11 -07:00
2026-05-21 19:17:51 -07:00
2026-04-20 03:07:32 -07:00
2026-03-04 17:23:23 +03:00
2026-04-16 19:39:21 -07:00
2026-04-19 16:53:11 -07:00
2026-05-09 17:53:35 -07:00
2026-04-04 16:57:24 -07:00
2026-04-19 16:47:20 -07:00
2026-04-08 21:37:51 -07:00
2026-04-20 23:20:33 -07:00
2026-05-28 00:19:31 -07:00
2026-05-09 14:46:34 -07:00
2026-03-29 21:51:44 -07:00
2026-04-09 16:24:53 -07:00
2026-05-27 00:43:32 -07:00
2026-04-30 20:38:09 -07:00
2026-04-08 17:23:15 -07:00
2026-05-25 14:52:24 -07:00
2026-05-23 02:20:14 -07:00
2026-04-11 16:22:50 -07:00
2026-05-28 00:19:31 -07:00
2026-04-23 21:51:19 -07:00
2026-05-09 18:44:58 -07:00
2026-05-25 01:41:19 -07:00
2026-05-22 03:33:01 -07:00
2026-05-25 01:41:19 -07:00
2026-05-25 05:50:29 -07:00
2026-05-22 03:33:01 -07:00
2026-04-30 02:53:20 -07:00
2026-05-15 12:11:32 -07:00
2026-04-16 14:23:16 -07:00
2026-05-13 17:32:22 -07:00
2026-04-21 17:49:39 -07:00
2026-05-11 23:02:15 -07:00
2026-05-18 22:29:45 -07:00
2026-05-25 05:15:55 -07:00
2026-04-30 02:53:20 -07:00
2026-05-24 18:04:54 -07:00
2026-05-13 22:04:28 -07:00
2026-05-25 14:30:06 -07:00
2026-05-18 10:51:15 -07:00
2026-05-04 00:04:36 +05:30
2026-05-13 16:39:41 -07:00
2026-05-13 16:39:41 -07:00
2026-05-21 16:40:04 +05:30
2026-05-12 13:59:22 -04:00
2026-04-20 00:32:09 -07:00
2026-05-21 14:17:39 -07:00
2026-05-27 13:59:12 +10:00
2026-04-25 06:41:58 -07:00
2026-05-21 16:40:04 +05:30
2026-05-21 16:40:04 +05:30
2026-05-21 16:40:04 +05:30
2026-05-19 19:27:34 -07:00
2026-05-21 16:40:04 +05:30
2026-05-13 22:31:28 -07:00
2026-05-21 16:40:04 +05:30
2026-05-21 16:40:04 +05:30
2026-03-01 01:54:27 +03:00
2026-05-27 18:26:44 -07:00
2026-05-21 16:40:04 +05:30
2026-05-21 02:38:45 +05:30
2026-05-25 03:35:33 -07:00
2026-05-19 17:27:24 -07:00