feat: initial Phase 1 implementation — core framework + Docker
- Data types (SManga, SChapter, Page, MangasPage, all Filter variants) - Source interfaces (Source, CatalogueSource) with MD5-based ID generation matching Tachiyomi/Suwayomi - HTTP client with per-host rate limiting, cookie jar, and 429 retry - FlareSolverr v1 client (FLARESOLVERR_URL env) - Generic GraphQL POST helper - goquery HTML parser wrappers - Source registry (panics on duplicate ID) - Multi-stage Dockerfile (golang:1.26-alpine + distroless) and compose.yml (postgres, flaresolverr, app)
This commit is contained in:
achmad
@@ -0,0 +1,170 @@
|
||||
# Phase 1 — Core Framework
|
||||
|
||||
All foundational types, interfaces, and helpers that every source implementation depends on.
|
||||
Nothing in `sources/` can be written until this phase is complete.
|
||||
|
||||
Reference:
|
||||
- Tachiyomi source contract: `/Users/achmad/Documents/Belajar/Android/extensions-source/core/src/` (HttpSource, CatalogueSource)
|
||||
- Kotlin originals: `/Users/achmad/Documents/Belajar/Android/extensions-source/lib-multisrc/madara/src/eu/kanade/tachiyomi/multisrc/madara/Madara.kt`
|
||||
- Suwayomi source interface: `/Users/achmad/Documents/Belajar/Web/Suwayomi-Server/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/`
|
||||
|
||||
---
|
||||
|
||||
## 1.1 Data Types — `internal/source/types.go`
|
||||
|
||||
- [x] `SManga` struct
|
||||
- [x] `URL`, `Title`, `Artist`, `Author`, `Description` string fields
|
||||
- [x] `Genre` string (comma-separated, matches Tachiyomi convention)
|
||||
- [x] `Status` int — 0=unknown, 1=ongoing, 2=completed, 3=licensed, 5=hiatus, 6=cancelled
|
||||
- [x] `ThumbnailURL` string
|
||||
- [x] `Initialized` bool (false until `GetMangaDetails` has been called)
|
||||
- [x] `SChapter` struct
|
||||
- [x] `URL`, `Name`, `Scanlator` string fields
|
||||
- [x] `DateUpload` int64 — unix milliseconds (matches Tachiyomi)
|
||||
- [x] `ChapterNumber` float32
|
||||
- [x] `Page` struct
|
||||
- [x] `Index` int
|
||||
- [x] `URL` string — page HTML URL or chapter URL
|
||||
- [x] `ImageURL` string — direct image URL (may be empty until `GetImageURL` resolves it)
|
||||
- [x] `MangasPage` struct
|
||||
- [x] `Mangas []SManga`
|
||||
- [x] `HasNextPage bool`
|
||||
- [x] Filter types
|
||||
- [x] `Filter` interface with `Name()` and `Value()` methods
|
||||
- [x] `TextFilter` — free-text input
|
||||
- [x] `CheckboxFilter` — boolean
|
||||
- [x] `TriStateFilter` — ignore / include / exclude (0/1/2)
|
||||
- [x] `SelectFilter` — dropdown with named options (name + values []string)
|
||||
- [x] `SortFilter` — sortable list with ascending/descending state
|
||||
- [x] `GroupFilter` — container of sub-filters
|
||||
- [x] Status constants (`StatusUnknown`, `StatusOngoing`, etc.)
|
||||
|
||||
---
|
||||
|
||||
## 1.2 Source Interfaces — `internal/source/interfaces.go`
|
||||
|
||||
- [x] `Source` interface (base)
|
||||
- [x] `ID() int64`
|
||||
- [x] `Name() string`
|
||||
- [x] `Lang() string`
|
||||
- [x] `CatalogueSource` interface (embeds Source)
|
||||
- [x] `SupportsLatest() bool`
|
||||
- [x] `GetPopularManga(page int) (MangasPage, error)`
|
||||
- [x] `GetLatestUpdates(page int) (MangasPage, error)`
|
||||
- [x] `GetSearchManga(page int, query string, filters []Filter) (MangasPage, error)`
|
||||
- [x] `GetMangaDetails(manga SManga) (SManga, error)`
|
||||
- [x] `GetChapterList(manga SManga) ([]SChapter, error)`
|
||||
- [x] `GetPageList(chapter SChapter) ([]Page, error)`
|
||||
- [x] `GetImageURL(page Page) (string, error)` — no-op default for sources that embed image URLs directly in pages
|
||||
- [x] `GetFilterList() []Filter`
|
||||
- [x] ID generation helper
|
||||
- [x] `GenerateSourceID(name, lang string) int64` — MD5 of `"${name.lowercase()}/$lang/1"`, first 8 bytes big-endian, sign bit cleared. Matches Suwayomi `HttpSource.generateId`. (The original plan incorrectly described Java `String.hashCode()`.)
|
||||
- [x] Unit test against computed IDs (MangaDex/en, MangaDex/all, HeanCms/en)
|
||||
|
||||
---
|
||||
|
||||
## 1.3 HTTP Client — `internal/httpclient/client.go`
|
||||
|
||||
- [x] `Client` struct wrapping `*http.Client`
|
||||
- [x] Persistent `cookiejar.Jar` (one per source instance)
|
||||
- [x] Per-host `rate.Limiter` map (keyed by `Host`) with `sync.Mutex`
|
||||
- [x] Configurable `RateLimit` (requests/sec) and `Burst` per source
|
||||
- [x] Configurable `Timeout` (default 30s)
|
||||
- [x] Default `User-Agent` header (Chrome/Android UA, matches Tachiyomi)
|
||||
- [x] `NewClient(opts ...Option) *Client` constructor with functional options
|
||||
- [x] `Do(req *http.Request) (*http.Response, error)` — applies rate limit before executing
|
||||
- [x] `Get(ctx context.Context, url string, headers map[string]string) (*http.Response, error)` helper
|
||||
- [x] `Post(ctx context.Context, url string, body io.Reader, contentType string, headers map[string]string) (*http.Response, error)` helper
|
||||
- [x] 429 retry logic
|
||||
- [x] Read `Retry-After` header (seconds or HTTP-date)
|
||||
- [x] Sleep and retry up to N times (configurable, default 3)
|
||||
- [x] Optional `Referer` injection (configurable per client)
|
||||
|
||||
---
|
||||
|
||||
## 1.4 FlareSolverr — `internal/httpclient/flaresolverr.go`
|
||||
|
||||
- [x] `FlareSolverrClient` struct
|
||||
- [x] `endpoint` string (FlareSolverr v1 base URL, e.g. `http://localhost:8191`)
|
||||
- [x] Underlying `*http.Client` for talking to FlareSolverr itself
|
||||
- [x] `Get(ctx context.Context, url string, cookies []Cookie) (html string, cookies []Cookie, err error)`
|
||||
- [x] POST `{"cmd":"request.get","url":"...","maxTimeout":60000}` to `/v1`
|
||||
- [x] Extract `solution.response` (rendered HTML body)
|
||||
- [x] Extract `solution.cookies` → convert to `[]*http.Cookie`
|
||||
- [x] Cookie reuse strategy — cookies returned to caller; caller injects into source's cookie jar
|
||||
- [x] `FlareSolverrResponse` JSON struct (full response shape)
|
||||
- [x] Config: `FLARESOLVERR_URL` env var (disable if empty — `NewFlareSolverrClient` returns error)
|
||||
|
||||
---
|
||||
|
||||
## 1.5 GraphQL Helper — `internal/httpclient/graphql.go`
|
||||
|
||||
Used only to talk to upstream sources that expose GraphQL (mangahub, senkuro, allanime, luscious, stashapp).
|
||||
Our own API is plain REST.
|
||||
|
||||
- [x] `GraphQLRequest` struct — `Query string`, `Variables any`
|
||||
- [x] `Post[T any](ctx context.Context, client *http.Client, url string, req GraphQLRequest, headers map[string]string) (T, error)`
|
||||
- [x] Marshal request to JSON
|
||||
- [x] POST with `Content-Type: application/json`
|
||||
- [x] Unmarshal `data` field of response into T
|
||||
- [x] Surface `errors[]` as a Go error if present
|
||||
- [x] `graphQLResponse[T any]` envelope struct
|
||||
|
||||
---
|
||||
|
||||
## 1.6 Header Builders — `internal/httpclient/headers.go`
|
||||
|
||||
- [x] `AndroidUA() string` — Chrome/Android user-agent string (matches Tachiyomi default)
|
||||
- [x] `DesktopUA() string` — desktop Chrome user-agent (for sources requiring desktop)
|
||||
- [x] `JSONHeaders() map[string]string` — `Content-Type: application/json`, `Accept: application/json`
|
||||
- [x] `FormHeaders() map[string]string` — `Content-Type: application/x-www-form-urlencoded`
|
||||
- [x] `WithRefererHeader(headers map[string]string, referer string) map[string]string`
|
||||
- [x] `WithOrigin(headers map[string]string, origin string) map[string]string`
|
||||
|
||||
---
|
||||
|
||||
## 1.7 HTML Parser — `internal/parser/html.go`
|
||||
|
||||
Thin wrappers over `github.com/PuerkitoBio/goquery` (Go equivalent of JSoup used in Tachiyomi extensions).
|
||||
|
||||
- [x] `Parse(html string) (*goquery.Document, error)` — parse raw HTML string
|
||||
- [x] `ParseResponse(resp *http.Response) (*goquery.Document, error)` — parse from HTTP response body
|
||||
- [x] `Select(doc *goquery.Document, css string) *goquery.Selection`
|
||||
- [x] `SelectFrom(sel *goquery.Selection, css string) *goquery.Selection`
|
||||
- [x] `Attr(sel *goquery.Selection, name string) string` — returns empty string if not found
|
||||
- [x] `AbsURL(sel *goquery.Selection, attr string, baseURL string) string` — resolves relative URLs
|
||||
- [x] `OwnText(sel *goquery.Selection) string` — text of element excluding child elements
|
||||
- [x] `TextTrim(sel *goquery.Selection) string` — `.Text()` with strings.TrimSpace
|
||||
- [x] `First(sel *goquery.Selection) *goquery.Selection`
|
||||
- [x] `Each(sel *goquery.Selection, fn func(i int, s *goquery.Selection))` — convenience wrapper
|
||||
|
||||
---
|
||||
|
||||
## 1.8 Registry — `internal/registry/registry.go`
|
||||
|
||||
- [x] Package-level `map[int64]source.CatalogueSource` protected by `sync.RWMutex`
|
||||
- [x] `Register(s source.CatalogueSource)` — panics on duplicate ID (caught at startup)
|
||||
- [x] `Get(id int64) (source.CatalogueSource, bool)`
|
||||
- [x] `All() []source.CatalogueSource` — returns sorted slice (by ID) for deterministic listing
|
||||
- [x] Each source package must call `registry.Register(NewXxx())` in its `init()` function
|
||||
- [ ] All source packages blank-imported in `cmd/server/main.go` (done when sources exist)
|
||||
|
||||
---
|
||||
|
||||
## 1.9 Docker — `Dockerfile` + `compose.yml`
|
||||
|
||||
- [x] `Dockerfile` — multi-stage build (`golang:1.26-alpine` builder + `distroless/static-debian12` runtime)
|
||||
- [x] `compose.yml` — services: `app`, `postgres`, `flaresolverr`
|
||||
- [x] `postgres` — official image, healthcheck, named volume
|
||||
- [x] `flaresolverr` — ghcr.io/flaresolverr/flaresolverr, depends on nothing
|
||||
- [x] `app` — builds from `Dockerfile`, depends on postgres healthcheck, env vars wired
|
||||
|
||||
---
|
||||
|
||||
## Checklist: Phase 1 Done When
|
||||
|
||||
- [x] `go build ./internal/...` succeeds with no errors
|
||||
- [x] `GenerateSourceID` matches Tachiyomi IDs for at least 3 known sources
|
||||
- [ ] `FlareSolverrClient.Get` returns rendered HTML for a Cloudflare-protected URL (manual test — requires running FlareSolverr)
|
||||
- [ ] `GraphQLPost` works against a public GraphQL endpoint (manual test)
|
||||
- [x] Registry panics on duplicate source ID (unit test)
|
||||
Reference in New Issue
Block a user