From 6953aa7833f32dae5ccd62b1f5f8afa5e0877d5a Mon Sep 17 00:00:00 2001 From: achmad Date: Thu, 14 May 2026 13:23:39 +0700 Subject: [PATCH] docs: add HTTP Client Architecture section to phase4-standalone Document the unified httpclient.Client design, FlareSolverr integration, cookie jar sharing, and Kotlin vs Go mapping. --- docs/phase4-standalone.md | 112 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 107 insertions(+), 5 deletions(-) diff --git a/docs/phase4-standalone.md b/docs/phase4-standalone.md index 0634935..a7bbfdf 100755 --- a/docs/phase4-standalone.md +++ b/docs/phase4-standalone.md @@ -29,6 +29,108 @@ Reference: - `/Users/achmad/Documents/Belajar/Android/extensions-source/src/all/` - `/Users/achmad/Documents/Belajar/Android/extensions-source/src/en/` +## HTTP Client Architecture + +### Cookie Jar + +The `httpclient.Client` creates a `cookiejar.Jar` for every instance. All cookies from +direct HTTP responses and from FlareSolverr are stored in this shared jar. Sources can +read cookies from the jar using the `Cookie(name, host)` method: + +```go +// After a request, check the jar for a specific cookie (like Kotlin's +// client.cookieJar.loadForRequest()): +value := client.Cookie("mhub_access", "mangahub.io") +``` + +### Response Headers from FlareSolverr + +When `Do()` falls back to FlareSolverr, the `doFS()` method now properly propagates +the actual response headers from FlareSolverr (including `Set-Cookie`) instead of +copying request headers into the response. Cookies from FS are also explicitly added +as `Set-Cookie` headers and fed into the shared cookie jar. + +### Single Unified Client + +All sources share one unified HTTP client (`internal/httpclient.Client`) that handles both +direct requests and Cloudflare/DDoS bypass transparently: + +``` +httpclient.DefaultClient() + ├── Direct HTTP (net/http + cookie jar + rate limiter) + └── FlareSolverr fallback (auto-detected from FLARESOLVERR_URL env var) +``` + +The `Do(req)` method implements **adaptive logic** matching the Kotlin `cloudflareClient`: + +1. **Try direct** — normal HTTP request with shared cookie jar + rate limiting +2. **If 403/503** (Cloudflare/DDoS challenge) — falls back to **FlareSolverr raw mode** +3. **FlareSolverr solves the challenge** — returns actual server response + cookies +4. **Cookies fed into shared jar** — subsequent requests to the same host skip FS +5. **Chrome HTML wrapper stripped** — FlareSolverr wraps responses in + `...
body
` — the client + strips this wrapper so callers receive the actual server body (JSON or HTML). + +### Kotlin vs Go Mapping + +| Kotlin (`cloudflareClient`) | Go (`httpclient.Client`) | +|---|---| +| `OkHttpClient` with Cloudflare interceptor | `net/http.Client` with FlareSolverr fallback | +| Intercepts 503 → solves via WebView → retries with cookies | Tries direct first → on 403/503 → FS raw → strips wrapper | +| Returns raw server response body | Returns stripped FS body or direct body | +| Shared across all sources | `DefaultClient()` singleton shared across all sources | + +### How Sources Use the Client + +```go +import "goyomi/internal/httpclient" + +// Default: use shared singleton (preferred for most sources) +func (s *Source) fetch(ctx context.Context) error { + req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + req.Header.Set("Accept", "application/json") + resp, err := httpclient.DefaultClient().Do(req) + // ... +} + +// Custom rate limit: create a dedicated client +func New() *Source { + c := httpclient.NewClient(httpclient.WithRateLimit(5, 10)) + return &Source{client: c} +} + +// Custom headers: set on the request (no need for a custom client) +func (s *Source) fetch(ctx context.Context) error { + req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + req.Header.Set("Accept", "text/css") // DDOS-Guard bypass + resp, err := httpclient.DefaultClient().Do(req) +} +``` + +### FlareSolverr Integration + +FlareSolverr is auto-configured from the `FLARESOLVERR_URL` environment variable. +If unset, the client works in direct-only mode (no Cloudflare bypass). + +The `flare` package (`internal/httpclient/flare/`) is a backward-compatible shim — +all it does is alias `httpclient.Client` and `httpclient.NewClient`. Sources that +imported `flare` before continue to compile without changes. + +### Design Decisions + +1. **Why a singleton?** — Shared cookie jar means cookies from FlareSolverr (solved + challenges) benefit all sources. Shared rate limiter prevents hammering the same host. +2. **Why strip FS wrapper?** — FlareSolverr routes requests through Chrome, which wraps + JSON responses in `...
JSON
...`. Without stripping, JSON parsers + fail. For HTML responses, Chrome adds `` tags but the page content + remains parseable by goquery. +3. **Why direct first?** — Most sites don't have Cloudflare. Direct requests are faster + (no Chrome overhead) and return clean response bodies. FS is only invoked on actual + challenge responses (403/503). +4. **Why not Chrome rendering?** — The Kotlin `cloudflareClient` does NOT render through + Chrome. It intercepts 503, solves the challenge, and returns the raw server response. + Our approach matches this behavior. + Detailed implementation notes for complex sources are in the **Notes** section at the bottom. --- @@ -607,12 +709,12 @@ Detailed implementation notes for complex sources are in the **Notes** section a ## Notes — Complex Sources -### Project Requirement: HTTP Client Selection -When porting sources from Kotlin to Go, ALWAYS check the Kotlin reference to determine whether the source needs: -- **Normal HTTP client** (`goyomi/internal/httpclient`) - for REST APIs returning JSON -- **FlareSolverr client** (`goyomi/internal/httpclient/flare`) - for JavaScript-rendered pages (uses `network.cloudflareClient` in Kotlin) +### HTTP Client Selection +All sources use the same unified client — `httpclient.DefaultClient()`. You no longer need +to decide between direct HTTP and FlareSolverr; the client handles both automatically. -Check in Kotlin: `override val client = network.cloudflareClient` or `network.cloudflareClient.newBuilder()` → needs FlareSolverr +In Kotlin, `override val client = network.cloudflareClient` signals the source needs +Cloudflare bypass — but in Go, the adaptive `Do()` method provides it transparently. ### `all/mangadex` ⚠️ - Rate limit: 5 req/s (`golang.org/x/time/rate`)