Files

T

achmad 2db6aad209 chore: register 4 new sources in main.go and checklist

2026-05-14 22:37:16 +07:00

25 KiB

Executable File

Raw Blame History

Phase 4 — Standalone Sources

Complete port checklist. Check a box when the source passes a basic smoke test (popular/latest list returns ≥1 result, or detail+pages resolve for a known URL).

After implementing each source: mark its checkbox [x] in this doc and add it to the import list in cmd/server/main.go.

Important: Source Types

Base Sources (`lib-multisrc`)

Shared implementations in:

/Users/achmad/Documents/Belajar/Android/extensions-source/lib-multisrc/
/Users/achmad/Documents/Belajar/Web/goyomi/sources/base/

These are the core implementations that standalone sources may use.

Standalone Sources (`all/` and `en/`)

Individual extensions in:

/Users/achmad/Documents/Belajar/Android/extensions-source/src/all/
/Users/achmad/Documents/Belajar/Android/extensions-source/src/en/

These may or may not use a base source from lib-multisrc — and even when they do, they often add their own implementation on top (overriding methods, adding extra parsing, custom filters, etc.). When porting, check if the Kotlin source extends a base class (e.g., MangaReader, Madara, ComicGeek) — if so, use the corresponding Go base implementation as the foundation, then layer the source-specific overrides. Otherwise, implement fully standalone.

Reference:

/Users/achmad/Documents/Belajar/Android/extensions-source/src/all/
/Users/achmad/Documents/Belajar/Android/extensions-source/src/en/

HTTP Client Architecture

The HTTP client uses a hybrid approach: httpcloak for Chrome TLS fingerprint emulation, with FlareSolverr as a fallback for challenge solving.

TLS Fingerprint Problem

Go's net/http has a different TLS fingerprint (JA3/JA4 ciphers, HTTP/2 settings) than Chrome. When Go sends a cf_clearance cookie obtained from FlareSolverr's Chrome, Cloudflare rejects it because the TLS fingerprint doesn't match Chrome's. Every direct request gets re-challenged.

The fix: use httpcloak as the transport, which mimics Chrome's TLS fingerprint perfectly. Now when FlareSolverr solves a challenge and returns cf_clearance, the cookie is fed into httpcloak's session. Subsequent requests via httpcloak (with matching TLS fingerprint + cookie) pass Cloudflare without re-challenge.

Architecture

httpclient.DefaultClient()
  ├── httpcloak.Session (Chrome TLS fingerprint + cookie jar)
  │     ├── 200 → return response (fast path, no re-challenge)
  │     └── 403/503 → FlareSolverr fallback
  └── FlareSolverr raw mode (auto-detected from FLARESOLVERR_URL env var)
        └── Cookies fed back into httpcloak session for next request

Flows

Non-Cloudflare site: httpcloak → 200. Fast, ~0.5s.
Cloudflare site, first request: httpcloak → 403 → FS solves challenge (~12-60s) → cf_clearance cookie stored in httpcloak session.
Cloudflare site, subsequent requests: httpcloak (with Chrome TLS + clearance cookie) → 200. Fast, ~1-2s.

FlareSolverr Integration

FlareSolverr is auto-configured from FLARESOLVERR_URL env var (e.g., http://localhost:8191). If unset, the client works in direct-only mode.

Env Var	Default	Description
`FLARESOLVERR_URL`	—	FlareSolverr endpoint
`FLARESOLVERR_SESSION`	`goyomi`	FS browser session ID; reuses one Chrome instance. Set empty for per-request sessions (more parallel, more memory)
`FLARESOLVERR_LOG_LEVEL`	`info`	FS log verbosity

The flare package (internal/httpclient/flare/) is a backward-compatible alias for httpclient.Client. Sources that already import flare continue to compile.

Kotlin vs Go Mapping

Kotlin (`cloudflareClient`)	Go (`httpclient.Client`)
Android WebView (Chrome) → solves 503 → retries with cookies	httpcloak (Chrome TLS fingerprint) → on 403 → FS fallback → cookies fed to httpcloak
WebView + OkHttp share Android's TLS stack	httpcloak mimics Chrome's JA3 fingerprint
Challenge solved once, cookies reused	Challenge solved once, cookies reused via httpcloak session
Returns raw server response	Returns FS raw body or httpcloak body
Shared across all extensions	`DefaultClient()` singleton shared across all sources

Design Decisions

Why httpcloak? — Go's net/http TLS fingerprint doesn't match Chrome's, so Cloudflare clearance cookies from FS are rejected on direct requests. httpcloak emulates Chrome's JA3/JA4 fingerprint, making subsequent requests pass Cloudflare.
Why not go-cfscraper? — The pure-Go JS challenge solver (goja) and external runtimes (Node.js) both lack browser DOM APIs (location, document, window) that Cloudflare challenge scripts require. The DOM shims are fragile and break when Cloudflare updates. FlareSolverr with real Chrome is the only reliable solver.
Why FlareSolverr at all? — The first request to a Cloudflare site always gets challenged (no cf_clearance cookie yet). FlareSolverr's real Chrome solves it.
Why a singleton? — Shared httpcloak session = cookies from FS (solved challenges) benefit all sources. Shared rate limiter prevents hammering the same host.
Why strip FS wrapper? — FS routes requests through Chrome, which wraps JSON responses in <html>…<pre>…</pre>…</html>. Without stripping, JSON parsers fail.
Why FS session matters — With FLARESOLVERR_SESSION, FS reuses one Chrome instance. After the first challenge, cf_clearance is cached. Subsequent requests to the same domain are near-instant. Without a session, each request spawns a fresh Chrome, solving the challenge from scratch every time (~13s each).

Known goquery Limitation — `:has()` + Attribute Selectors

goquery (via the cascadia CSS engine) does not support :has() combined with attribute selectors like a[href*=/video/]. This works in Jsoup (Kotlin) but silently returns 0 matches in Go.

Wrong (CSS-only, works in Kotlin, returns 0 in Go):

doc.Find("figure:not(:has(a[href*=/video/]))")

Correct (programmatic filtering):

doc.Find("figure").Each(func(_ int, el *goquery.Selection) {
    if hasAttr(el, "a", "href", "/video/") { return }
    // process entry
})

func hasAttr(el *goquery.Selection, tag, attr, substr string) bool {
    found := false
    el.Find(tag).EachWithBreak(func(_ int, a *goquery.Selection) bool {
        if v, ok := a.Attr(attr); ok && strings.Contains(v, substr) {
            found = true; return false
        }
        return true
    })
    return found
}

Always check Kotlin references for :has() selectors and convert them to programmatic filtering when porting.

Detailed implementation notes for complex sources are in the Notes section at the bottom.

`sources/all/` — 125 sources

all/ahottie
all/akuma
all/allporncomicsco
all/asmhentai
all/baobua
all/beauty3600000
all/buondua
all/comicfury
all/comicgrowl
all/comicklive
all/comicskingdom ⚠️ no Kotlin reference found — skip
all/comicsvalley
all/comikey
all/commitstrip
all/coomer
all/coronaex
all/cosplaytele
all/cubari
all/danbooru ⚠️ see notes
all/deviantart
all/dragonballmultiverse
all/e621 ⚠️ see notes
all/elitebabes
all/everiaclub
all/everiaclubcom
all/femjoyhunter
all/foamgirl
all/foolslidecustomizable
all/fourkhd
all/ftvhunter
all/fuwayomi ⚠️ skip — server-side Fuwayomi integration, not a scraper
all/globalcomix ⚠️ see notes
all/grabberzone
all/hdoujin
all/hennojin
all/hentai3
all/hentaicosplay
all/hentaienvy
all/hentaiera
all/hentaifox
all/hentaihand
all/hentairox
all/hentaizap
all/hniscantrad
all/holonometria
all/honeytoon
all/imhentai
all/izneo
all/jjcos
all/joymiihub
all/junmeitu
all/kemono ⚠️ see notes
all/kiutaku
all/kodokustudio
all/koharu
all/komga ⚠️ see notes
all/lanraragi
all/leagueoflegends
all/lunaranime
all/luscious ⚠️ see notes
all/magicaltranslators
all/manga18me
all/mangaball
all/mangacrazy
all/mangadex ⚠️ see notes
all/mangadraft
all/mangafire
all/mangaforfree
all/mangaplus ⚠️ see notes
all/mangapluscreators
all/mangataro
all/mangatoon
all/mangaup
all/mango
all/manhuarm
all/manhwa18cc
all/manhwa18net
all/manhwa18uncensored
all/manhwaclubnet
all/manhwadashraw
all/mayotune
all/metarthunter
all/miauscan
all/misskon
all/mitaku
all/myreadingmanga
all/namicomi
all/nhentaicom ⚠️ see notes
all/nhentaixxx
all/niadd
all/ninemanga
all/novelcool
all/ososedki
all/pandachaika
all/peppercarrot
all/photos18
all/pixiv ⚠️ see notes
all/playmatehunter
all/pornpics
all/projectsuki
all/qtoon
all/rokuhentai
all/sakuramanhwa
all/sandraandwoo
all/seraphicdeviltry
all/simplycosplay
all/simplyhentai
all/stashapp ⚠️ see notes
all/taddyink
all/tappytoon
all/thelibraryofohara
all/thunderscans
all/toomics
all/twicomi
all/uncensoredmanhwa
all/vinnieVeritas
all/webtoons ⚠️ see notes
all/xarthunter
all/xasiatalbums
all/xgmn
all/xinmeitulu
all/xiutaku
all/xkcd
all/yabai
all/yaoimangaonline
all/yellownote
all/yskcomics

`sources/en/` — 430 sources

en/akaicomic
en/alandal
en/allanime ⚠️ see notes
en/allporncomic
en/allporncomicio
en/anisascans
en/apcomics
en/aquamanga
en/arcrelight → base: MangAdventure
en/arenascans
en/armageddon
en/artlapsa
en/arvencomics
en/arvenscans
en/aryascans
en/asiatoon
en/asmotoon
en/assortedscans
en/asurascans ⚠️ see notes
en/athreascans
en/atsumaru → standalone HttpSource
en/aurora → standalone HttpSource
en/azcomic → standalone HttpSource
en/azuki
en/baektoons
en/bakkin ⚠️ see notes
en/bakkinselfhosted
en/batcave
en/battleinfivesecondsaftermeeting
en/beehentai → base: MadTheme
en/bookwalker ⚠️ see notes
en/boratscans → base: Madara
en/boxmanhwa → base: MadTheme
en/broccolisoup → standalone HttpSource
en/bunmanga → base: Madara
en/buttsmithy → standalone HttpSource
en/clonemanga → standalone HttpSource
en/clowncorps → standalone HttpSource
en/cmanhua
en/cocomic
en/coffeemanga → base: Madara (complex)
en/collectedcurios
en/comicasura
en/comiccx
en/comichubfree
en/comickfan
en/comickiba
en/comicland
en/comicsland
en/comix
en/crowscans
en/cucumbermanga → base: Madara
en/culturedworks
en/cutiecomics
en/dankefurslesen
en/darklegacycomics
en/darkscans
en/darkscience
en/darthsdroids
en/deathtollscans
en/decadencescans → base: Madara
en/dflowscans
en/digitalcomicmuseum
en/divascans
en/doujinio
en/doujins
en/dragontea
en/drakescans
en/dynasty
en/eggporncomics
en/egscomics
en/eighteenporncomic
en/eightmuses
en/elanschool
en/elftoon
en/epicmanga
en/erisscans
en/ero18x
en/erofus
en/erosscans
en/evascans
en/evilflowers
en/existentialcomics
en/explosm
en/ezmanga
en/fablescans
en/fairyscans
en/firescans → base: Madara (complex)
en/flamecomics
en/frierenonline
en/gakamangas
en/galaxydegenscans
en/galaxymanga
en/gedecomix
en/gingertoon
en/goda
en/gourmetscans
en/greedscans
en/grimscans
en/grrlpower
en/gunnerkriggcourt
en/guya ⚠️ see notes
en/gwtb
en/hachirumi
en/hadesscans
en/harimanga
en/hentai3zcc
en/hentai4free
en/hentaidex
en/hentaihere
en/hentaikun
en/hentainexus
en/hentairead → base: Madara (complex)
en/hentaireadio
en/hentaisco
en/hentaixcomic
en/hentaixdickgirl
en/hentaixyuri
en/hentara
en/heytoon
en/hijalascans
en/hiperdex
en/hiveworks
en/hm2d
en/honkaiimpact
en/hotcomics
en/hyakuro
en/infernalvoidscans
en/infinityscans
en/irovedout
en/isekaiscantop
en/jinmangas
en/jnovel
en/kagane
en/kaizenscan
en/kaliscancom → base: MadTheme
en/kaliscanio → base: MadTheme
en/kaliscanme
en/kappabeast
en/kaynscans
en/keenspot
en/kenscans
en/kewnscans
en/killsixbilliondemons
en/kingcomix
en/kingofshojo
en/kissmangain
en/kmanga
en/kodansha
en/ksgroupscans
en/kunmanga
en/kuramanga
en/lagoonscans
en/leslievictims
en/lhtranslation
en/likemanga
en/likemangain
en/lilymanga
en/linkmanga → base: Madara
en/loadingartist
en/luascans
en/luminaretranslations
en/lunatoons
en/madaradex
en/madarascans
en/madokami
en/magusmanga
en/mahouirexnohentaikarte
en/manga18club
en/manga18free
en/manga18fx
en/manga18x
en/mangabat
en/mangablaze
en/mangabolt
en/mangabtt
en/mangabuddy → base: MadTheme
en/mangabuddyme → base: MadTheme
en/mangaclash
en/mangacloud
en/mangacute → base: MadTheme
en/mangadass
en/mangade
en/mangademon
en/mangadia
en/mangadistrict
en/mangadotnet
en/mangadrama
en/mangafab → base: MadTheme
en/mangaforest → base: MadTheme
en/mangaforfreecom
en/mangafox
en/mangafoxfun → base: MangaHub
en/mangafreak
en/mangafree
en/mangagg
en/mangago
en/mangagofun
en/mangahe
en/mangahen
en/mangahentai
en/mangahere
en/mangahereonl → base: MangaHub
en/mangahubio → base: MangaHub
en/mangaka
en/mangakakalot
en/mangakakalotfun → base: MangaHub
en/mangakatana
en/mangakiss
en/mangamaniacs
en/mangamo
en/mangamob
en/mangamonk → base: MadTheme
en/manganel → base: MangaHub
en/manganelo
en/manganow
en/mangaonlinefun → base: MangaHub
en/mangaowlio
en/mangapandaonl → base: MangaHub
en/mangapill
en/mangapuma
en/mangarawclub
en/mangaread
en/mangareadercc
en/mangareadersite → base: MangaHub
en/mangareadorg
en/mangasaga → base: MadTheme
en/mangasect
en/mangaspin → base: MadTheme
en/mangasushi
en/mangatellers
en/mangatoday → base: MangaHub
en/mangatown
en/mangatrend
en/mangatx
en/mangaxyz → base: MadTheme
en/manhuafast
en/manhuafastnet
en/manhuahot
en/manhuanext
en/manhuanow → base: MadTheme
en/manhuaplus
en/manhuaplusorg
en/manhuarush
en/manhuascanus
en/manhuasite → base: MadTheme
en/manhuatop
en/manhuaus
en/manhuazonghe
en/manhwa18
en/manhwa18org
en/manhwa68
en/manhwabuddy
en/manhwaclan
en/manhwacomics
en/manhwaden
en/manhwaget
en/manhwahub
en/manhwajoy
en/manhwalike
en/manhwalover
en/manhwamanhua
en/manhwaread
en/manhwareads
en/manhwatoon
en/manhwatop
en/manhwax
en/manhwaxxl
en/manhwaz
en/manhwazone
en/manta
en/megatokyo
en/mehgazone
en/meitoon
en/mgjinx
en/mgreadio
en/milftoon
en/mistscans
en/mlbblore
en/monochromecustom
en/monochromescans
en/multporn
en/murimscan
en/myhentaicomics
en/myhentaigallery
en/necroscans
en/newmanhwa
en/nexcomic
en/nikatoons
en/nineanime
en/ninehentai
en/ninekon
en/novel24h
en/novelcrow
en/noxenscans
en/nuxscans
en/nyanukafe
en/nyrascans
en/nyxscans
en/octopusmanga
en/oglaf
en/ohjoysextoy
en/omegascans
en/onemangaco → base: MangaHub
en/onemangainfo → base: MangaHub
en/onepunchmanonline
en/onlythebesthentai
en/oots
en/oppaistream
en/orchisasia
en/orionscans
en/paradisescans
en/paragonscans
en/paritehaber
en/patchfriday
en/pawmanga
en/petrotechsociety
en/philiascans
en/plutoscans
en/pmscans
en/porncomix
en/qiscans
en/questionablecontent
en/ragescans
en/randowiz
en/ravenscans
en/razure
en/rdscans
en/readallcomicscom
en/readattackontitanshingekinokyojinmanga
en/readberserkmanga
en/readblackclovermangaonline
en/readbokunoheroacademiamyheroacademiamanga
en/readchainsawmanmangaonline
en/readcomiconline
en/readcomicsonline
en/readfairytailedenszeromangaonline
en/readhaikyuumangaonline
en/readjujutsukaisenmangaonline
en/readkingdommangaonline
en/readnanatsunotaizai7deadlysinsmangaonline
en/readnarutoborutosamurai8mangaonline
en/readonepiecemangaonline
en/readonepunchmanmangaonlinetwo
en/readsololevelingmangamanhwaonline
en/readtokyoghoulretokyoghoulmangaonline
en/readvagabondmanga
en/reallifecomics
en/reimanga
en/renascans
en/resetscans
en/restscans
en/retsu
en/revivalscans
en/rinkocomics
en/ritharscans
en/rizzcomic
en/rizzcomicunoriginal
en/rokaricomics
en/roliascan
en/rosesquadscans
en/ryumanga
en/s2manga
en/sanascans
en/saturdaymorningbreakfastcomics
en/schlockmercenary
en/setsuscans
en/shibamanga
en/shojoscans
en/sirenscans
en/skymanga
en/sleepytranslations → base: Madara
en/solarandsundry
en/spmanhwa
en/spyfakku
en/stonescape
en/sunshinebutterflyscans
en/supermega
en/suryascans
en/swordscomic
en/tapastic
en/tcbscans
en/tcbscansunoriginal
en/teamshadowi
en/templescan
en/theblank
en/theduckwebcomics
en/thepropertyofhate
en/timelesstoons
en/todaymanga
en/toon18
en/toongod
en/toonily
en/toonilyme
en/toonitube
en/toonizy
en/topmanhua
en/topmanhuafan
en/topmanhuanet
en/tritiniascans
en/utoon
en/valirscans
en/vanillascans
en/vgperson
en/vizshonenjump
en/voyceme
en/vyvymanga
en/vyvymangaorg
en/warforrayuba
en/wearehunger
en/webcomics
en/webdexscans
en/webnovel
en/webtoonscan
en/webtoonxyz
en/weebcentral
en/whalemanga
en/witchscans
en/woopread
en/writerscans
en/wuxiaworld
en/xlecx
en/xomanga
en/xoxocomics
en/yakshacomics
en/yaoihot
en/yaoihub
en/yaoiscan
en/yaoitoon
en/yorai
en/zazamanga
en/zinchanmanga
en/zinchanmangacom
en/zinmanga
en/zinmanganet

Notes — Complex Sources

HTTP Client Selection

All sources use the same unified client — httpclient.DefaultClient(). You no longer need to decide between direct HTTP and FlareSolverr; the client handles both automatically.

In Kotlin, override val client = network.cloudflareClient signals the source needs Cloudflare bypass — but in Go, the adaptive Do() method provides it transparently.

`all/mangadex` ⚠️

Rate limit: 5 req/s (golang.org/x/time/rate)
GetPopularManga — GET /manga?order[rating]=desc&includes[]=cover_art&limit=20&offset={(n-1)*20}
GetLatestUpdates — GET /manga?order[updatedAt]=desc&includes[]=cover_art
GetSearchManga — tag filters (includedTags[], excludedTags[]), demographic, content rating, status, sort
GetMangaDetails — GET /manga/{id}?includes[]=cover_art&includes[]=author&includes[]=artist; cover URL from relationships
GetChapterList — GET /manga/{id}/feed?translatedLanguage[]=en&limit=500; paginate until all chapters fetched
GetPageList — GET /at-home/server/{chapterId} → {baseUrl}/{quality}/{hash}/{filename}; quality = data or data-saver
GetFilterList — GET /manga/tag; cache tag list
Language configurable via MANGADEX_LANG env var (default en)

`all/nhentaicom` ⚠️

FlareSolverr required
GetPageList — extract JSON blob from document.getElementById(...) in <script> tag; reconstruct URLs as {imageServer}/galleries/{mediaId}/{pageNum}.{ext}

`all/kemono` ⚠️

FlareSolverr required
Service + creator = manga; posts = chapters; attachment files = pages
File URLs: prefix relative paths with {base}/data
GetChapterList — paginate in 50-post increments via ?o={offset}

`all/komga` ⚠️

Self-hosted; KOMGA_BASE_URL, KOMGA_USERNAME, KOMGA_PASSWORD env vars; Basic Auth
Series = manga; Books = chapters; pages via GET /api/v1/books/{bookId}/pages/{n}/thumbnail

`all/e621` ⚠️

Optional Basic Auth (E621_USERNAME, E621_API_KEY); required for Gold content
Pools = manga; posts = pages; GET /pools.json, GET /posts.json?tags=pool:{id}

`all/danbooru` ⚠️

Optional Basic Auth; free accounts limited to 2 tags per search
Pools = manga; pool post_ids = pages; image URL from file_url field

`all/pixiv` ⚠️

Auth: PIXIV_PHPSESSID env var → Cookie header
GetPageList — GET /ajax/illust/{illustId}/pages; extract urls.original
Must set Referer: https://www.pixiv.net/ on all image requests or get 403

`all/luscious` ⚠️

GraphQL POST to https://members.luscious.net/graphql/playground
Albums = manga; SearchAlbumPictures paginated for page list

`all/mangaplus` ⚠️

JSON endpoint: GET https://jumpg-webapi.tokyo-cdn.com/api/title_detail?title_id={id}
GetPageList — XOR page URL decryption using per-page encryptionKey (hex string); implement in GetImageURL
Required header: Origin: https://mangaplus.shueisha.co.jp

`all/stashapp` ⚠️

Self-hosted; STASHAPP_BASE_URL, STASHAPP_API_KEY env vars
GraphQL POST to {base}/graphql; ApiKey header

`all/globalcomix` ⚠️

Signed CDN URLs with short expiry — do NOT cache image_url in DB; always fetch fresh page URLs

`all/webtoons` ⚠️

HMAC-SHA256 Sec-Webtoon-Client-Data header: HMAC-SHA256(fixedKey, "{url}_{timestamp}") → base64; fixed key in extension source
Web API: https://global.apis.naver.com/webtoonSvc/v1/...

`en/allanime` ⚠️

GraphQL POST to https://api.allanime.day/api
Episode-based chapters; episodeString used as chapter number
Multiple CDN quality tiers in page response; pick highest

`en/asurascans` ⚠️

FlareSolverr required
Page selector varies by layout version; check extension source for current selector

`en/bakkin` ⚠️

No list/search; all manga from a single JSON URL; enumerate from object keys
GetSearchManga does client-side title filtering only

`en/bookwalker` ⚠️

DRM-protected; metadata only
GetPageList returns empty slice; pages not accessible without purchase

`en/guya` ⚠️

GET {base}/api/get_all_series/ returns all manga at once; no pagination
Scanlation group filter applied client-side in response parsing

25 KiB Executable File Raw Blame History

Phase 4 — Standalone Sources

Important: Source Types

Base Sources (lib-multisrc)

Standalone Sources (all/ and en/)

HTTP Client Architecture

TLS Fingerprint Problem

Architecture

Flows

FlareSolverr Integration

Kotlin vs Go Mapping

Design Decisions

Known goquery Limitation — :has() + Attribute Selectors

sources/all/ — 125 sources

sources/en/ — 430 sources

Notes — Complex Sources

HTTP Client Selection

all/mangadex ⚠️

all/nhentaicom ⚠️

all/kemono ⚠️

all/komga ⚠️

all/e621 ⚠️

all/danbooru ⚠️

all/pixiv ⚠️

all/luscious ⚠️

all/mangaplus ⚠️

all/stashapp ⚠️

all/globalcomix ⚠️

all/webtoons ⚠️

en/allanime ⚠️

en/asurascans ⚠️

en/bakkin ⚠️

en/bookwalker ⚠️

en/guya ⚠️