docs: add HTTP Client Architecture section to phase4-standalone
Document the unified httpclient.Client design, FlareSolverr integration, cookie jar sharing, and Kotlin vs Go mapping.
This commit is contained in:
achmad
+107
-5
@@ -29,6 +29,108 @@ Reference:
|
||||
- `/Users/achmad/Documents/Belajar/Android/extensions-source/src/all/`
|
||||
- `/Users/achmad/Documents/Belajar/Android/extensions-source/src/en/`
|
||||
|
||||
## HTTP Client Architecture
|
||||
|
||||
### Cookie Jar
|
||||
|
||||
The `httpclient.Client` creates a `cookiejar.Jar` for every instance. All cookies from
|
||||
direct HTTP responses and from FlareSolverr are stored in this shared jar. Sources can
|
||||
read cookies from the jar using the `Cookie(name, host)` method:
|
||||
|
||||
```go
|
||||
// After a request, check the jar for a specific cookie (like Kotlin's
|
||||
// client.cookieJar.loadForRequest()):
|
||||
value := client.Cookie("mhub_access", "mangahub.io")
|
||||
```
|
||||
|
||||
### Response Headers from FlareSolverr
|
||||
|
||||
When `Do()` falls back to FlareSolverr, the `doFS()` method now properly propagates
|
||||
the actual response headers from FlareSolverr (including `Set-Cookie`) instead of
|
||||
copying request headers into the response. Cookies from FS are also explicitly added
|
||||
as `Set-Cookie` headers and fed into the shared cookie jar.
|
||||
|
||||
### Single Unified Client
|
||||
|
||||
All sources share one unified HTTP client (`internal/httpclient.Client`) that handles both
|
||||
direct requests and Cloudflare/DDoS bypass transparently:
|
||||
|
||||
```
|
||||
httpclient.DefaultClient()
|
||||
├── Direct HTTP (net/http + cookie jar + rate limiter)
|
||||
└── FlareSolverr fallback (auto-detected from FLARESOLVERR_URL env var)
|
||||
```
|
||||
|
||||
The `Do(req)` method implements **adaptive logic** matching the Kotlin `cloudflareClient`:
|
||||
|
||||
1. **Try direct** — normal HTTP request with shared cookie jar + rate limiting
|
||||
2. **If 403/503** (Cloudflare/DDoS challenge) — falls back to **FlareSolverr raw mode**
|
||||
3. **FlareSolverr solves the challenge** — returns actual server response + cookies
|
||||
4. **Cookies fed into shared jar** — subsequent requests to the same host skip FS
|
||||
5. **Chrome HTML wrapper stripped** — FlareSolverr wraps responses in
|
||||
`<html><head>...<meta...></head><body><pre>body</pre></body></html>` — the client
|
||||
strips this wrapper so callers receive the actual server body (JSON or HTML).
|
||||
|
||||
### Kotlin vs Go Mapping
|
||||
|
||||
| Kotlin (`cloudflareClient`) | Go (`httpclient.Client`) |
|
||||
|---|---|
|
||||
| `OkHttpClient` with Cloudflare interceptor | `net/http.Client` with FlareSolverr fallback |
|
||||
| Intercepts 503 → solves via WebView → retries with cookies | Tries direct first → on 403/503 → FS raw → strips wrapper |
|
||||
| Returns raw server response body | Returns stripped FS body or direct body |
|
||||
| Shared across all sources | `DefaultClient()` singleton shared across all sources |
|
||||
|
||||
### How Sources Use the Client
|
||||
|
||||
```go
|
||||
import "goyomi/internal/httpclient"
|
||||
|
||||
// Default: use shared singleton (preferred for most sources)
|
||||
func (s *Source) fetch(ctx context.Context) error {
|
||||
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
resp, err := httpclient.DefaultClient().Do(req)
|
||||
// ...
|
||||
}
|
||||
|
||||
// Custom rate limit: create a dedicated client
|
||||
func New() *Source {
|
||||
c := httpclient.NewClient(httpclient.WithRateLimit(5, 10))
|
||||
return &Source{client: c}
|
||||
}
|
||||
|
||||
// Custom headers: set on the request (no need for a custom client)
|
||||
func (s *Source) fetch(ctx context.Context) error {
|
||||
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
req.Header.Set("Accept", "text/css") // DDOS-Guard bypass
|
||||
resp, err := httpclient.DefaultClient().Do(req)
|
||||
}
|
||||
```
|
||||
|
||||
### FlareSolverr Integration
|
||||
|
||||
FlareSolverr is auto-configured from the `FLARESOLVERR_URL` environment variable.
|
||||
If unset, the client works in direct-only mode (no Cloudflare bypass).
|
||||
|
||||
The `flare` package (`internal/httpclient/flare/`) is a backward-compatible shim —
|
||||
all it does is alias `httpclient.Client` and `httpclient.NewClient`. Sources that
|
||||
imported `flare` before continue to compile without changes.
|
||||
|
||||
### Design Decisions
|
||||
|
||||
1. **Why a singleton?** — Shared cookie jar means cookies from FlareSolverr (solved
|
||||
challenges) benefit all sources. Shared rate limiter prevents hammering the same host.
|
||||
2. **Why strip FS wrapper?** — FlareSolverr routes requests through Chrome, which wraps
|
||||
JSON responses in `<html><head>...<pre>JSON</pre>...`. Without stripping, JSON parsers
|
||||
fail. For HTML responses, Chrome adds `<meta charset>` tags but the page content
|
||||
remains parseable by goquery.
|
||||
3. **Why direct first?** — Most sites don't have Cloudflare. Direct requests are faster
|
||||
(no Chrome overhead) and return clean response bodies. FS is only invoked on actual
|
||||
challenge responses (403/503).
|
||||
4. **Why not Chrome rendering?** — The Kotlin `cloudflareClient` does NOT render through
|
||||
Chrome. It intercepts 503, solves the challenge, and returns the raw server response.
|
||||
Our approach matches this behavior.
|
||||
|
||||
Detailed implementation notes for complex sources are in the **Notes** section at the bottom.
|
||||
|
||||
---
|
||||
@@ -607,12 +709,12 @@ Detailed implementation notes for complex sources are in the **Notes** section a
|
||||
|
||||
## Notes — Complex Sources
|
||||
|
||||
### Project Requirement: HTTP Client Selection
|
||||
When porting sources from Kotlin to Go, ALWAYS check the Kotlin reference to determine whether the source needs:
|
||||
- **Normal HTTP client** (`goyomi/internal/httpclient`) - for REST APIs returning JSON
|
||||
- **FlareSolverr client** (`goyomi/internal/httpclient/flare`) - for JavaScript-rendered pages (uses `network.cloudflareClient` in Kotlin)
|
||||
### HTTP Client Selection
|
||||
All sources use the same unified client — `httpclient.DefaultClient()`. You no longer need
|
||||
to decide between direct HTTP and FlareSolverr; the client handles both automatically.
|
||||
|
||||
Check in Kotlin: `override val client = network.cloudflareClient` or `network.cloudflareClient.newBuilder()` → needs FlareSolverr
|
||||
In Kotlin, `override val client = network.cloudflareClient` signals the source needs
|
||||
Cloudflare bypass — but in Go, the adaptive `Do()` method provides it transparently.
|
||||
|
||||
### `all/mangadex` ⚠️
|
||||
- Rate limit: 5 req/s (`golang.org/x/time/rate`)
|
||||
|
||||
Reference in New Issue
Block a user