The challenge detection only needs a sample (32KB), but when the body
is not a challenge, the full body must be returned. The previous code
discarded everything after 32KB, causing Madara parsers to find 0
manga items in truncated HTML.
Some anti-bot challenge pages return HTTP 200 instead of 403/503,
bypassing the FS fallback. Read response body on 200, check for
challenge indicators, and fall through to FlareSolverr if detected.
- Use httpcloak.Session (Chrome JA3/JA4 fingerprint) as primary transport
- Adaptive: direct request via httpcloak first; FlareSolverr fallback on 403/503
- FS cookies fed into httpcloak session so subsequent requests reuse
cf_clearance (Chrome fingerprint + cookie = no re-challenge)
- FlareSolverr timeout increased to 120s for slow challenges
- Sanitize FS cookie values (strip quotes/newlines to avoid Go cookie warnings)
- Remove go-cfscraper dependency (pure JS solver was fragile)
Re-enables the direct-then-FS approach removed earlier due to TLS
fingerprint mismatch. With FS sessions configured, the fallback path
is fast after the first request per domain (Chrome caches cf_clearance).
Non-Cloudflare sites still benefit from direct HTTP speed.
- Guard isCloudflareChallenge with directStatus >= 400 to prevent
overriding status to 0 when no direct request was made
- When FS returns challenge page without a prior direct status,
return an error instead of silently passing HTTP 0
- Restore default FS session ID to 'goyomi' — without a session,
each request spawns a new Chrome, causing timeouts under load
- Add Message field to FlareSolverrResponse for better error reporting
- Document FLARESOLVERR_SESSION env var: shared session = fast after
1st request, but serializes. Set empty for parallel (resource-heavy).
When -v is passed, test-sources.sh passes it through to go test -v.
sourcetest.Run uses testing.Verbose() to print the full manga list
from GetPopularManga and GetLatestUpdates, showing title + URL.
- Remove global ProxyEnabled() logic from httpclient
- Each source now explicitly chooses client at import time:
- flare client: for JS-rendering/cloudflare sources
- normal httpclient: for REST API sources
- Updated 29 base sources based on Kotlin reference (network.cloudflareClient)
- Add config table for storing FlareSolverr proxy setting
- Add HTTP endpoints to get/set proxy status (GET/POST /api/config/flaresolverr)
- Refactor httpclient to support proxy mode (requests go through FlareSolverr)
- Add verbose logging for debugging
- Add POST support to FlareSolverr client
Usage:
GET /api/config/flaresolverr - returns {flaresolverr_proxy: bool}
POST /api/config/flaresolverr - body: {enabled: true/false}
achmad