feat: initial Phase 1 implementation — core framework + Docker

- Data types (SManga, SChapter, Page, MangasPage, all Filter variants) - Source interfaces (Source, CatalogueSource) with MD5-based ID generation matching Tachiyomi/Suwayomi - HTTP client with per-host rate limiting, cookie jar, and 429 retry - FlareSolverr v1 client (FLARESOLVERR_URL env) - Generic GraphQL POST helper - goquery HTML parser wrappers - Source registry (panics on duplicate ID) - Multi-stage Dockerfile (golang:1.26-alpine + distroless) and compose.yml (postgres, flaresolverr, app)
2026-05-10 21:23:24 +07:00
commit 85d2ea6143
23 changed files with 2864 additions and 0 deletions
@@ -0,0 +1,183 @@
+# Phase 2 — Database Layer
+
+Persistent storage using PostgreSQL via pgx. Migrations run on startup. sqlc generates type-safe query code.
+
+Reference schema modeled after Suwayomi-Server:
+- `/Users/achmad/Documents/Belajar/Web/Suwayomi-Server/server/src/main/kotlin/suwayomi/tachidesk/manga/model/table/`
+- Migrations: `/Users/achmad/Documents/Belajar/Web/Suwayomi-Server/server/src/main/kotlin/suwayomi/tachidesk/server/database/migration/`
+
+---
+
+## 2.1 Schema Migration — `internal/db/migrations/001_init.sql`
+
+- [ ] `sources` table
+  - [ ] `id BIGINT PRIMARY KEY` — generated via `GenerateSourceID(name, lang)` same as Tachiyomi
+  - [ ] `name VARCHAR(128) NOT NULL`
+  - [ ] `lang VARCHAR(32) NOT NULL`
+  - [ ] `is_nsfw BOOLEAN NOT NULL DEFAULT FALSE`
+- [ ] `manga` table
+  - [ ] `id SERIAL PRIMARY KEY`
+  - [ ] `source_id BIGINT NOT NULL REFERENCES sources(id)`
+  - [ ] `url VARCHAR(2048) NOT NULL`
+  - [ ] `title VARCHAR(512) NOT NULL`
+  - [ ] `initialized BOOLEAN NOT NULL DEFAULT FALSE`
+  - [ ] `artist TEXT`, `author TEXT`, `description TEXT`, `genre TEXT`
+  - [ ] `status INTEGER NOT NULL DEFAULT 0`
+  - [ ] `thumbnail_url VARCHAR(2048)`
+  - [ ] `thumbnail_last_fetched BIGINT NOT NULL DEFAULT 0`
+  - [ ] `in_library BOOLEAN NOT NULL DEFAULT FALSE`
+  - [ ] `in_library_at BIGINT NOT NULL DEFAULT 0`
+  - [ ] `real_url VARCHAR(2048)`
+  - [ ] `last_fetched_at BIGINT NOT NULL DEFAULT 0`
+  - [ ] `chapters_last_fetched_at BIGINT NOT NULL DEFAULT 0`
+  - [ ] `update_strategy VARCHAR(64) NOT NULL DEFAULT 'ALWAYS_UPDATE'`
+  - [ ] `UNIQUE (source_id, url)`
+- [ ] `chapters` table
+  - [ ] `id SERIAL PRIMARY KEY`
+  - [ ] `manga_id INTEGER NOT NULL REFERENCES manga(id) ON DELETE CASCADE`
+  - [ ] `url VARCHAR(2048) NOT NULL`
+  - [ ] `name VARCHAR(512) NOT NULL`
+  - [ ] `date_upload BIGINT NOT NULL DEFAULT 0`
+  - [ ] `chapter_number REAL NOT NULL DEFAULT -1`
+  - [ ] `scanlator VARCHAR(256)`
+  - [ ] `source_order INTEGER NOT NULL`
+  - [ ] `is_read BOOLEAN NOT NULL DEFAULT FALSE`
+  - [ ] `is_bookmarked BOOLEAN NOT NULL DEFAULT FALSE`
+  - [ ] `last_page_read INTEGER NOT NULL DEFAULT 0`
+  - [ ] `last_read_at BIGINT NOT NULL DEFAULT 0`
+  - [ ] `fetched_at BIGINT NOT NULL DEFAULT 0`
+  - [ ] `real_url VARCHAR(2048)`
+  - [ ] `is_downloaded BOOLEAN NOT NULL DEFAULT FALSE`
+  - [ ] `page_count INTEGER NOT NULL DEFAULT -1`
+  - [ ] `UNIQUE (manga_id, url)`
+- [ ] `pages` table
+  - [ ] `id SERIAL PRIMARY KEY`
+  - [ ] `chapter_id INTEGER NOT NULL REFERENCES chapters(id) ON DELETE CASCADE`
+  - [ ] `"index" INTEGER NOT NULL`
+  - [ ] `url VARCHAR(2048) NOT NULL`
+  - [ ] `image_url TEXT`
+- [ ] `source_meta` table
+  - [ ] `source_id BIGINT NOT NULL REFERENCES sources(id)`
+  - [ ] `key VARCHAR(256) NOT NULL`
+  - [ ] `value TEXT NOT NULL`
+  - [ ] `PRIMARY KEY (source_id, key)`
+- [ ] Indexes
+  - [ ] `CREATE INDEX ON manga (source_id)`
+  - [ ] `CREATE INDEX ON manga (last_fetched_at)`
+  - [ ] `CREATE INDEX ON chapters (manga_id)`
+  - [ ] `CREATE INDEX ON pages (chapter_id)`
+
+---
+
+## 2.2 DB Initialization — `internal/db/db.go`
+
+- [ ] `pgxpool.Pool` init from `DATABASE_URL` env var
+  - [ ] Configurable `MaxConns` (default 10)
+  - [ ] Configurable `MinConns` (default 2)
+  - [ ] Connection health check on startup
+- [ ] Migration runner using `golang-migrate/migrate`
+  - [ ] Source: `iofs` (embed migration SQL files with `//go:embed`)
+  - [ ] Driver: `pgx5`
+  - [ ] Run `migrate.Up()` on startup; log version before/after
+  - [ ] Non-fatal on "no change" (already at latest)
+- [ ] `Queries` struct wrapping sqlc-generated query clients
+- [ ] `Close()` method to drain pool on shutdown
+
+---
+
+## 2.3 SQL Queries — `internal/db/queries/`
+
+Write `.sql` files, then run `sqlc generate` to produce type-safe Go.
+
+### `manga.sql`
+
+- [ ] `UpsertManga` — INSERT ... ON CONFLICT (source_id, url) DO UPDATE all fields; returns row
+- [ ] `GetMangaBySourceURL` — SELECT by source_id + url
+- [ ] `GetMangaByID` — SELECT by primary key
+- [ ] `ListMangaBySource` — SELECT by source_id ORDER BY last_fetched_at DESC
+- [ ] `UpdateMangaDetails` — UPDATE artist/author/description/genre/status/thumbnail/initialized for a manga id
+- [ ] `UpdateMangaFetchedAt` — UPDATE last_fetched_at = $now WHERE id = $id
+- [ ] `UpdateChaptersFetchedAt` — UPDATE chapters_last_fetched_at WHERE id = $id
+
+### `chapter.sql`
+
+- [ ] `UpsertChapter` — INSERT ... ON CONFLICT (manga_id, url) DO UPDATE name/date/chapter_number/scanlator/source_order
+- [ ] `GetChapterByID`
+- [ ] `ListChaptersByManga` — ORDER BY source_order
+- [ ] `UpdateChapterFetchedAt`
+
+### `page.sql`
+
+- [ ] `UpsertPage` — INSERT ... ON CONFLICT (chapter_id, index) DO UPDATE url/image_url
+- [ ] `ListPagesByChapter` — ORDER BY index
+- [ ] `UpdatePageImageURL` — UPDATE image_url WHERE id = $id
+
+### `source.sql`
+
+- [ ] `UpsertSource` — INSERT ... ON CONFLICT (id) DO UPDATE name/lang/is_nsfw
+- [ ] `ListSources`
+- [ ] `GetSourceByID`
+- [ ] `GetSourceMeta` — SELECT value WHERE source_id = $id AND key = $key
+- [ ] `SetSourceMeta` — INSERT ... ON CONFLICT DO UPDATE value
+
+---
+
+## 2.4 sqlc Configuration — `sqlc.yaml`
+
+- [ ] `version: "2"`
+- [ ] engine: `postgresql`
+- [ ] schema path pointing to `internal/db/migrations/`
+- [ ] queries path pointing to `internal/db/queries/*.sql`
+- [ ] output package: `internal/db/queries`
+- [ ] `emit_json_tags: true`
+- [ ] `emit_db_tags: true`
+
+---
+
+## 2.5 Data Flow & Cache Logic
+
+Four upsert flows — each called from the API handler before returning data.
+
+### Manga List (`GetPopularManga` / `GetLatestUpdates` / `GetSearchManga`)
+
+- [ ] Check `manga.last_fetched_at`: if within TTL (default 10 min) and `?refresh=false`, return DB rows
+- [ ] Otherwise call source, then `UpsertManga` for each returned SManga
+- [ ] Update `last_fetched_at` on upserted rows
+- [ ] `UpsertSource` for the source record (idempotent on every list call)
+
+### Manga Detail (`GetMangaDetails`)
+
+- [ ] Check `manga.initialized`: if true and within TTL (default 1h), return DB row
+- [ ] Otherwise call source, then `UpdateMangaDetails` + set `initialized=true`
+
+### Chapter List (`GetChapterList`)
+
+- [ ] Check `manga.chapters_last_fetched_at`: if within TTL (default 10 min), return DB rows
+- [ ] Otherwise call source, then `UpsertChapter` for each SChapter (preserving `source_order` = slice index)
+- [ ] Update `chapters_last_fetched_at` on the manga row
+
+### Page List (`GetPageList`)
+
+- [ ] Check if pages exist for chapter in DB
+- [ ] Otherwise call source, then `UpsertPage` for each Page
+- [ ] If source needs a second call to resolve image URLs (via `GetImageURL`), call it and `UpdatePageImageURL`
+- [ ] Store resolved `image_url` — do NOT cache for sources with signed/expiring URLs (globalcomix, etc.)
+
+### TTL Configuration
+
+- [ ] `MANGA_LIST_TTL_SECONDS` env var (default 600)
+- [ ] `MANGA_DETAIL_TTL_SECONDS` env var (default 3600)
+- [ ] `CHAPTER_LIST_TTL_SECONDS` env var (default 600)
+- [ ] `?refresh=true` query param bypasses TTL on any route
+
+---
+
+## Checklist: Phase 2 Done When
+
+- [ ] `golang-migrate` runs `001_init.sql` on a fresh DB without error
+- [ ] `sqlc generate` completes without errors; generated files compile
+- [ ] `UpsertManga` + `GetMangaBySourceURL` round-trip test passes
+- [ ] `UpsertChapter` correctly sets `source_order` from slice position
+- [ ] TTL cache logic returns DB rows on second call (verify no source HTTP call via log)
+- [ ] `?refresh=true` bypasses TTL and re-fetches from source
+- [ ] All tables visible in `psql` after API calls