Files

T

achmad 6953aa7833 docs: add HTTP Client Architecture section to phase4-standalone

Document the unified httpclient.Client design, FlareSolverr integration,
cookie jar sharing, and Kotlin vs Go mapping.

2026-05-14 13:23:39 +07:00

24 KiB

Executable File

Raw Blame History

Phase 4 — Standalone Sources

Complete port checklist. Check a box when the source passes a basic smoke test (popular/latest list returns ≥1 result, or detail+pages resolve for a known URL).

After implementing each source: mark its checkbox [x] in this doc and add it to the import list in cmd/server/main.go.

Important: Source Types

Base Sources (`lib-multisrc`)

Shared implementations in:

/Users/achmad/Documents/Belajar/Android/extensions-source/lib-multisrc/
/Users/achmad/Documents/Belajar/Web/goyomi/sources/base/

These are the core implementations that standalone sources may use.

Standalone Sources (`all/` and `en/`)

Individual extensions in:

/Users/achmad/Documents/Belajar/Android/extensions-source/src/all/
/Users/achmad/Documents/Belajar/Android/extensions-source/src/en/

These may or may not use a base source from lib-multisrc — and even when they do, they often add their own implementation on top (overriding methods, adding extra parsing, custom filters, etc.). When porting, check if the Kotlin source extends a base class (e.g., MangaReader, Madara, ComicGeek) — if so, use the corresponding Go base implementation as the foundation, then layer the source-specific overrides. Otherwise, implement fully standalone.

Reference:

/Users/achmad/Documents/Belajar/Android/extensions-source/src/all/
/Users/achmad/Documents/Belajar/Android/extensions-source/src/en/

HTTP Client Architecture

The httpclient.Client creates a cookiejar.Jar for every instance. All cookies from direct HTTP responses and from FlareSolverr are stored in this shared jar. Sources can read cookies from the jar using the Cookie(name, host) method:

// After a request, check the jar for a specific cookie (like Kotlin's
// client.cookieJar.loadForRequest()):
value := client.Cookie("mhub_access", "mangahub.io")

Response Headers from FlareSolverr

When Do() falls back to FlareSolverr, the doFS() method now properly propagates the actual response headers from FlareSolverr (including Set-Cookie) instead of copying request headers into the response. Cookies from FS are also explicitly added as Set-Cookie headers and fed into the shared cookie jar.

Single Unified Client

All sources share one unified HTTP client (internal/httpclient.Client) that handles both direct requests and Cloudflare/DDoS bypass transparently:

httpclient.DefaultClient()
  ├── Direct HTTP (net/http + cookie jar + rate limiter)
  └── FlareSolverr fallback (auto-detected from FLARESOLVERR_URL env var)

The Do(req) method implements adaptive logic matching the Kotlin cloudflareClient:

Try direct — normal HTTP request with shared cookie jar + rate limiting
If 403/503 (Cloudflare/DDoS challenge) — falls back to FlareSolverr raw mode
FlareSolverr solves the challenge — returns actual server response + cookies
Cookies fed into shared jar — subsequent requests to the same host skip FS
Chrome HTML wrapper stripped — FlareSolverr wraps responses in <html><head>...<meta...></head><body><pre>body</pre></body></html> — the client strips this wrapper so callers receive the actual server body (JSON or HTML).

Kotlin vs Go Mapping

Kotlin (`cloudflareClient`)	Go (`httpclient.Client`)
`OkHttpClient` with Cloudflare interceptor	`net/http.Client` with FlareSolverr fallback
Intercepts 503 → solves via WebView → retries with cookies	Tries direct first → on 403/503 → FS raw → strips wrapper
Returns raw server response body	Returns stripped FS body or direct body
Shared across all sources	`DefaultClient()` singleton shared across all sources

How Sources Use the Client

import "goyomi/internal/httpclient"

// Default: use shared singleton (preferred for most sources)
func (s *Source) fetch(ctx context.Context) error {
    req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    req.Header.Set("Accept", "application/json")
    resp, err := httpclient.DefaultClient().Do(req)
    // ...
}

// Custom rate limit: create a dedicated client
func New() *Source {
    c := httpclient.NewClient(httpclient.WithRateLimit(5, 10))
    return &Source{client: c}
}

// Custom headers: set on the request (no need for a custom client)
func (s *Source) fetch(ctx context.Context) error {
    req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    req.Header.Set("Accept", "text/css")  // DDOS-Guard bypass
    resp, err := httpclient.DefaultClient().Do(req)
}

FlareSolverr Integration

FlareSolverr is auto-configured from the FLARESOLVERR_URL environment variable. If unset, the client works in direct-only mode (no Cloudflare bypass).

The flare package (internal/httpclient/flare/) is a backward-compatible shim — all it does is alias httpclient.Client and httpclient.NewClient. Sources that imported flare before continue to compile without changes.

Design Decisions

Why a singleton? — Shared cookie jar means cookies from FlareSolverr (solved challenges) benefit all sources. Shared rate limiter prevents hammering the same host.
Why strip FS wrapper? — FlareSolverr routes requests through Chrome, which wraps JSON responses in <html><head>...<pre>JSON</pre>.... Without stripping, JSON parsers fail. For HTML responses, Chrome adds <meta charset> tags but the page content remains parseable by goquery.
Why direct first? — Most sites don't have Cloudflare. Direct requests are faster (no Chrome overhead) and return clean response bodies. FS is only invoked on actual challenge responses (403/503).
Why not Chrome rendering? — The Kotlin cloudflareClient does NOT render through Chrome. It intercepts 503, solves the challenge, and returns the raw server response. Our approach matches this behavior.

Detailed implementation notes for complex sources are in the Notes section at the bottom.

`sources/all/` — 125 sources

all/ahottie
all/akuma
all/allporncomicsco
all/asmhentai
all/baobua
all/beauty3600000
all/buondua
all/comicfury
all/comicgrowl
all/comicklive
all/comicskingdom ⚠️ no Kotlin reference found — skip
all/comicsvalley
all/comikey
all/commitstrip
all/coomer
all/coronaex
all/cosplaytele
all/cubari
all/danbooru ⚠️ see notes
all/deviantart
all/dragonballmultiverse
all/e621 ⚠️ see notes
all/elitebabes
all/everiaclub
all/everiaclubcom
all/femjoyhunter
all/foamgirl
all/foolslidecustomizable
all/fourkhd
all/ftvhunter
all/fuwayomi ⚠️ skip — server-side Fuwayomi integration, not a scraper
all/globalcomix ⚠️ see notes
all/grabberzone
all/hdoujin
all/hennojin
all/hentai3
all/hentaicosplay
all/hentaienvy
all/hentaiera
all/hentaifox
all/hentaihand
all/hentairox
all/hentaizap
all/hniscantrad
all/holonometria
all/honeytoon
all/imhentai
all/izneo
all/jjcos
all/joymiihub
all/junmeitu
all/kemono ⚠️ see notes
all/kiutaku
all/kodokustudio
all/koharu
all/komga ⚠️ see notes
all/lanraragi
all/leagueoflegends
all/lunaranime
all/luscious ⚠️ see notes
all/magicaltranslators
all/manga18me
all/mangaball
all/mangacrazy
all/mangadex ⚠️ see notes
all/mangadraft
all/mangafire
all/mangaforfree
all/mangaplus ⚠️ see notes
all/mangapluscreators
all/mangataro
all/mangatoon
all/mangaup
all/mango
all/manhuarm
all/manhwa18cc
all/manhwa18net
all/manhwa18uncensored
all/manhwaclubnet
all/manhwadashraw
all/mayotune
all/metarthunter
all/miauscan
all/misskon
all/mitaku
all/myreadingmanga
all/namicomi
all/nhentaicom ⚠️ see notes
all/nhentaixxx
all/niadd
all/ninemanga
all/novelcool
all/ososedki
all/pandachaika
all/peppercarrot
all/photos18
all/pixiv ⚠️ see notes
all/playmatehunter
all/pornpics
all/projectsuki
all/qtoon
all/rokuhentai
all/sakuramanhwa
all/sandraandwoo
all/seraphicdeviltry
all/simplycosplay
all/simplyhentai
all/stashapp ⚠️ see notes
all/taddyink
all/tappytoon
all/thelibraryofohara
all/thunderscans
all/toomics
all/twicomi
all/uncensoredmanhwa
all/vinnieVeritas
all/webtoons ⚠️ see notes
all/xarthunter
all/xasiatalbums
all/xgmn
all/xinmeitulu
all/xiutaku
all/xkcd
all/yabai
all/yaoimangaonline
all/yellownote
all/yskcomics

`sources/en/` — 430 sources

en/akaicomic
en/alandal
en/allanime ⚠️ see notes
en/allporncomic
en/allporncomicio
en/anisascans
en/apcomics
en/aquamanga
en/arcrelight → base: MangAdventure
en/arenascans
en/armageddon
en/artlapsa
en/arvencomics
en/arvenscans
en/aryascans
en/asiatoon
en/asmotoon
en/assortedscans
en/asurascans ⚠️ see notes
en/athreascans
en/atsumaru → standalone HttpSource
en/aurora → standalone HttpSource
en/azcomic → standalone HttpSource
en/azuki
en/baektoons
en/bakkin ⚠️ see notes
en/bakkinselfhosted
en/batcave
en/battleinfivesecondsaftermeeting
en/beehentai → base: MadTheme
en/bookwalker ⚠️ see notes
en/boratscans → base: Madara
en/boxmanhwa → base: MadTheme
en/broccolisoup → standalone HttpSource
en/bunmanga → base: Madara
en/buttsmithy → standalone HttpSource
en/clonemanga → standalone HttpSource
en/clowncorps → standalone HttpSource
en/cmanhua
en/cocomic
en/coffeemanga → base: Madara (complex)
en/collectedcurios
en/comicasura
en/comiccx
en/comichubfree
en/comickfan
en/comickiba
en/comicland
en/comicsland
en/comix
en/crowscans
en/cucumbermanga → base: Madara
en/culturedworks
en/cutiecomics
en/dankefurslesen
en/darklegacycomics
en/darkscans
en/darkscience
en/darthsdroids
en/deathtollscans
en/decadencescans → base: Madara
en/dflowscans
en/digitalcomicmuseum
en/divascans
en/doujinio
en/doujins
en/dragontea
en/drakescans
en/dynasty
en/eggporncomics
en/egscomics
en/eighteenporncomic
en/eightmuses
en/elanschool
en/elftoon
en/epicmanga
en/erisscans
en/ero18x
en/erofus
en/erosscans
en/evascans
en/evilflowers
en/existentialcomics
en/explosm
en/ezmanga
en/fablescans
en/fairyscans
en/firescans → base: Madara (complex)
en/flamecomics
en/frierenonline
en/gakamangas
en/galaxydegenscans
en/galaxymanga
en/gedecomix
en/gingertoon
en/goda
en/gourmetscans
en/greedscans
en/grimscans
en/grrlpower
en/gunnerkriggcourt
en/guya ⚠️ see notes
en/gwtb
en/hachirumi
en/hadesscans
en/harimanga
en/hentai3zcc
en/hentai4free
en/hentaidex
en/hentaihere
en/hentaikun
en/hentainexus
en/hentairead → base: Madara (complex)
en/hentaireadio
en/hentaisco
en/hentaixcomic
en/hentaixdickgirl
en/hentaixyuri
en/hentara
en/heytoon
en/hijalascans
en/hiperdex
en/hiveworks
en/hm2d
en/honkaiimpact
en/hotcomics
en/hyakuro
en/infernalvoidscans
en/infinityscans
en/irovedout
en/isekaiscantop
en/jinmangas
en/jnovel
en/kagane
en/kaizenscan
en/kaliscancom → base: MadTheme
en/kaliscanio → base: MadTheme
en/kaliscanme
en/kappabeast
en/kaynscans
en/keenspot
en/kenscans
en/kewnscans
en/killsixbilliondemons
en/kingcomix
en/kingofshojo
en/kissmangain
en/kmanga
en/kodansha
en/ksgroupscans
en/kunmanga
en/kuramanga
en/lagoonscans
en/leslievictims
en/lhtranslation
en/likemanga
en/likemangain
en/lilymanga
en/linkmanga → base: Madara
en/loadingartist
en/luascans
en/luminaretranslations
en/lunatoons
en/madaradex
en/madarascans
en/madokami
en/magusmanga
en/mahouirexnohentaikarte
en/manga18club
en/manga18free
en/manga18fx
en/manga18x
en/mangabat
en/mangablaze
en/mangabolt
en/mangabtt
en/mangabuddy → base: MadTheme
en/mangabuddyme → base: MadTheme
en/mangaclash
en/mangacloud
en/mangacute → base: MadTheme
en/mangadass
en/mangade
en/mangademon
en/mangadia
en/mangadistrict
en/mangadotnet
en/mangadrama
en/mangafab → base: MadTheme
en/mangaforest → base: MadTheme
en/mangaforfreecom
en/mangafox
en/mangafoxfun → base: MangaHub
en/mangafreak
en/mangafree
en/mangagg
en/mangago
en/mangagofun
en/mangahe
en/mangahen
en/mangahentai
en/mangahere
en/mangahereonl → base: MangaHub
en/mangahubio → base: MangaHub
en/mangaka
en/mangakakalot
en/mangakakalotfun → base: MangaHub
en/mangakatana
en/mangakiss
en/mangamaniacs
en/mangamo
en/mangamob
en/mangamonk → base: MadTheme
en/manganel → base: MangaHub
en/manganelo
en/manganow
en/mangaonlinefun → base: MangaHub
en/mangaowlio
en/mangapandaonl → base: MangaHub
en/mangapill
en/mangapuma
en/mangarawclub
en/mangaread
en/mangareadercc
en/mangareadersite → base: MangaHub
en/mangareadorg
en/mangasaga → base: MadTheme
en/mangasect
en/mangaspin → base: MadTheme
en/mangasushi
en/mangatellers
en/mangatoday → base: MangaHub
en/mangatown
en/mangatrend
en/mangatx
en/mangaxyz → base: MadTheme
en/manhuafast
en/manhuafastnet
en/manhuahot
en/manhuanext
en/manhuanow → base: MadTheme
en/manhuaplus
en/manhuaplusorg
en/manhuarush
en/manhuascanus
en/manhuasite → base: MadTheme
en/manhuatop
en/manhuaus
en/manhuazonghe
en/manhwa18
en/manhwa18org
en/manhwa68
en/manhwabuddy
en/manhwaclan
en/manhwacomics
en/manhwaden
en/manhwaget
en/manhwahub
en/manhwajoy
en/manhwalike
en/manhwalover
en/manhwamanhua
en/manhwaread
en/manhwareads
en/manhwatoon
en/manhwatop
en/manhwax
en/manhwaxxl
en/manhwaz
en/manhwazone
en/manta
en/megatokyo
en/mehgazone
en/meitoon
en/mgjinx
en/mgreadio
en/milftoon
en/mistscans
en/mlbblore
en/monochromecustom
en/monochromescans
en/multporn
en/murimscan
en/myhentaicomics
en/myhentaigallery
en/necroscans
en/newmanhwa
en/nexcomic
en/nikatoons
en/nineanime
en/ninehentai
en/ninekon
en/novel24h
en/novelcrow
en/noxenscans
en/nuxscans
en/nyanukafe
en/nyrascans
en/nyxscans
en/octopusmanga
en/oglaf
en/ohjoysextoy
en/omegascans
en/onemangaco → base: MangaHub
en/onemangainfo → base: MangaHub
en/onepunchmanonline
en/onlythebesthentai
en/oots
en/oppaistream
en/orchisasia
en/orionscans
en/paradisescans
en/paragonscans
en/paritehaber
en/patchfriday
en/pawmanga
en/petrotechsociety
en/philiascans
en/plutoscans
en/pmscans
en/porncomix
en/qiscans
en/questionablecontent
en/ragescans
en/randowiz
en/ravenscans
en/razure
en/rdscans
en/readallcomicscom
en/readattackontitanshingekinokyojinmanga
en/readberserkmanga
en/readblackclovermangaonline
en/readbokunoheroacademiamyheroacademiamanga
en/readchainsawmanmangaonline
en/readcomiconline
en/readcomicsonline
en/readfairytailedenszeromangaonline
en/readhaikyuumangaonline
en/readjujutsukaisenmangaonline
en/readkingdommangaonline
en/readnanatsunotaizai7deadlysinsmangaonline
en/readnarutoborutosamurai8mangaonline
en/readonepiecemangaonline
en/readonepunchmanmangaonlinetwo
en/readsololevelingmangamanhwaonline
en/readtokyoghoulretokyoghoulmangaonline
en/readvagabondmanga
en/reallifecomics
en/reimanga
en/renascans
en/resetscans
en/restscans
en/retsu
en/revivalscans
en/rinkocomics
en/ritharscans
en/rizzcomic
en/rizzcomicunoriginal
en/rokaricomics
en/roliascan
en/rosesquadscans
en/ryumanga
en/s2manga
en/sanascans
en/saturdaymorningbreakfastcomics
en/schlockmercenary
en/setsuscans
en/shibamanga
en/shojoscans
en/sirenscans
en/skymanga
en/sleepytranslations → base: Madara
en/solarandsundry
en/spmanhwa
en/spyfakku
en/stonescape
en/sunshinebutterflyscans
en/supermega
en/suryascans
en/swordscomic
en/tapastic
en/tcbscans
en/tcbscansunoriginal
en/teamshadowi
en/templescan
en/theblank
en/theduckwebcomics
en/thepropertyofhate
en/timelesstoons
en/todaymanga
en/toon18
en/toongod
en/toonily
en/toonilyme
en/toonitube
en/toonizy
en/topmanhua
en/topmanhuafan
en/topmanhuanet
en/tritiniascans
en/utoon
en/valirscans
en/vanillascans
en/vgperson
en/vizshonenjump
en/voyceme
en/vyvymanga
en/vyvymangaorg
en/warforrayuba
en/wearehunger
en/webcomics
en/webdexscans
en/webnovel
en/webtoonscan
en/webtoonxyz
en/weebcentral
en/whalemanga
en/witchscans
en/woopread
en/writerscans
en/wuxiaworld
en/xlecx
en/xomanga
en/xoxocomics
en/yakshacomics
en/yaoihot
en/yaoihub
en/yaoiscan
en/yaoitoon
en/yorai
en/zazamanga
en/zinchanmanga
en/zinchanmangacom
en/zinmanga
en/zinmanganet

Notes — Complex Sources

HTTP Client Selection

All sources use the same unified client — httpclient.DefaultClient(). You no longer need to decide between direct HTTP and FlareSolverr; the client handles both automatically.

In Kotlin, override val client = network.cloudflareClient signals the source needs Cloudflare bypass — but in Go, the adaptive Do() method provides it transparently.

`all/mangadex` ⚠️

Rate limit: 5 req/s (golang.org/x/time/rate)
GetPopularManga — GET /manga?order[rating]=desc&includes[]=cover_art&limit=20&offset={(n-1)*20}
GetLatestUpdates — GET /manga?order[updatedAt]=desc&includes[]=cover_art
GetSearchManga — tag filters (includedTags[], excludedTags[]), demographic, content rating, status, sort
GetMangaDetails — GET /manga/{id}?includes[]=cover_art&includes[]=author&includes[]=artist; cover URL from relationships
GetChapterList — GET /manga/{id}/feed?translatedLanguage[]=en&limit=500; paginate until all chapters fetched
GetPageList — GET /at-home/server/{chapterId} → {baseUrl}/{quality}/{hash}/{filename}; quality = data or data-saver
GetFilterList — GET /manga/tag; cache tag list
Language configurable via MANGADEX_LANG env var (default en)

`all/nhentaicom` ⚠️

FlareSolverr required
GetPageList — extract JSON blob from document.getElementById(...) in <script> tag; reconstruct URLs as {imageServer}/galleries/{mediaId}/{pageNum}.{ext}

`all/kemono` ⚠️

FlareSolverr required
Service + creator = manga; posts = chapters; attachment files = pages
File URLs: prefix relative paths with {base}/data
GetChapterList — paginate in 50-post increments via ?o={offset}

`all/komga` ⚠️

Self-hosted; KOMGA_BASE_URL, KOMGA_USERNAME, KOMGA_PASSWORD env vars; Basic Auth
Series = manga; Books = chapters; pages via GET /api/v1/books/{bookId}/pages/{n}/thumbnail

`all/e621` ⚠️

Optional Basic Auth (E621_USERNAME, E621_API_KEY); required for Gold content
Pools = manga; posts = pages; GET /pools.json, GET /posts.json?tags=pool:{id}

`all/danbooru` ⚠️

Optional Basic Auth; free accounts limited to 2 tags per search
Pools = manga; pool post_ids = pages; image URL from file_url field

`all/pixiv` ⚠️

Auth: PIXIV_PHPSESSID env var → Cookie header
GetPageList — GET /ajax/illust/{illustId}/pages; extract urls.original
Must set Referer: https://www.pixiv.net/ on all image requests or get 403

`all/luscious` ⚠️

GraphQL POST to https://members.luscious.net/graphql/playground
Albums = manga; SearchAlbumPictures paginated for page list

`all/mangaplus` ⚠️

JSON endpoint: GET https://jumpg-webapi.tokyo-cdn.com/api/title_detail?title_id={id}
GetPageList — XOR page URL decryption using per-page encryptionKey (hex string); implement in GetImageURL
Required header: Origin: https://mangaplus.shueisha.co.jp

`all/stashapp` ⚠️

Self-hosted; STASHAPP_BASE_URL, STASHAPP_API_KEY env vars
GraphQL POST to {base}/graphql; ApiKey header

`all/globalcomix` ⚠️

Signed CDN URLs with short expiry — do NOT cache image_url in DB; always fetch fresh page URLs

`all/webtoons` ⚠️

HMAC-SHA256 Sec-Webtoon-Client-Data header: HMAC-SHA256(fixedKey, "{url}_{timestamp}") → base64; fixed key in extension source
Web API: https://global.apis.naver.com/webtoonSvc/v1/...

`en/allanime` ⚠️

GraphQL POST to https://api.allanime.day/api
Episode-based chapters; episodeString used as chapter number
Multiple CDN quality tiers in page response; pick highest

`en/asurascans` ⚠️

FlareSolverr required
Page selector varies by layout version; check extension source for current selector

`en/bakkin` ⚠️

No list/search; all manga from a single JSON URL; enumerate from object keys
GetSearchManga does client-side title filtering only

`en/bookwalker` ⚠️

DRM-protected; metadata only
GetPageList returns empty slice; pages not accessible without purchase

`en/guya` ⚠️

GET {base}/api/get_all_series/ returns all manga at once; no pagination
Scanlation group filter applied client-side in response parsing

24 KiB Executable File Raw Blame History

Phase 4 — Standalone Sources

Important: Source Types

Base Sources (lib-multisrc)

Standalone Sources (all/ and en/)

HTTP Client Architecture

Cookie Jar

Response Headers from FlareSolverr

Single Unified Client

Kotlin vs Go Mapping

How Sources Use the Client

FlareSolverr Integration

Design Decisions

sources/all/ — 125 sources

sources/en/ — 430 sources

Notes — Complex Sources

HTTP Client Selection

all/mangadex ⚠️

all/nhentaicom ⚠️

all/kemono ⚠️

all/komga ⚠️

all/e621 ⚠️

all/danbooru ⚠️

all/pixiv ⚠️

all/luscious ⚠️

all/mangaplus ⚠️

all/stashapp ⚠️

all/globalcomix ⚠️

all/webtoons ⚠️

en/allanime ⚠️

en/asurascans ⚠️

en/bakkin ⚠️

en/bookwalker ⚠️

en/guya ⚠️