feat: Phase 2 — database layer

- PostgreSQL schema: sources, manga, chapters, pages, source_meta with indexes - golang-migrate runner with embedded SQL via go:embed (pgx5:// scheme) - sqlc-generated type-safe queries for all tables (pgx/v5 native) - Config package with all env vars including TTL durations - Wire DB init and config into cmd/server/main.go
2026-05-10 21:32:40 +07:00
parent 85d2ea6143
commit 95cab106d8
20 changed files with 1291 additions and 139 deletions
@@ -8,176 +8,153 @@ Reference schema modeled after Suwayomi-Server:

 ---

-## 2.1 Schema Migration — `internal/db/migrations/001_init.sql`
+## 2.1 Schema Migration — `internal/db/migrations/000001_init.up.sql`

- [ ] `sources` table
-  - [ ] `id BIGINT PRIMARY KEY` — generated via `GenerateSourceID(name, lang)` same as Tachiyomi
-  - [ ] `name VARCHAR(128) NOT NULL`
-  - [ ] `lang VARCHAR(32) NOT NULL`
-  - [ ] `is_nsfw BOOLEAN NOT NULL DEFAULT FALSE`
- [ ] `manga` table
-  - [ ] `id SERIAL PRIMARY KEY`
-  - [ ] `source_id BIGINT NOT NULL REFERENCES sources(id)`
-  - [ ] `url VARCHAR(2048) NOT NULL`
-  - [ ] `title VARCHAR(512) NOT NULL`
-  - [ ] `initialized BOOLEAN NOT NULL DEFAULT FALSE`
-  - [ ] `artist TEXT`, `author TEXT`, `description TEXT`, `genre TEXT`
-  - [ ] `status INTEGER NOT NULL DEFAULT 0`
-  - [ ] `thumbnail_url VARCHAR(2048)`
-  - [ ] `thumbnail_last_fetched BIGINT NOT NULL DEFAULT 0`
-  - [ ] `in_library BOOLEAN NOT NULL DEFAULT FALSE`
-  - [ ] `in_library_at BIGINT NOT NULL DEFAULT 0`
-  - [ ] `real_url VARCHAR(2048)`
-  - [ ] `last_fetched_at BIGINT NOT NULL DEFAULT 0`
-  - [ ] `chapters_last_fetched_at BIGINT NOT NULL DEFAULT 0`
-  - [ ] `update_strategy VARCHAR(64) NOT NULL DEFAULT 'ALWAYS_UPDATE'`
-  - [ ] `UNIQUE (source_id, url)`
- [ ] `chapters` table
-  - [ ] `id SERIAL PRIMARY KEY`
-  - [ ] `manga_id INTEGER NOT NULL REFERENCES manga(id) ON DELETE CASCADE`
-  - [ ] `url VARCHAR(2048) NOT NULL`
-  - [ ] `name VARCHAR(512) NOT NULL`
-  - [ ] `date_upload BIGINT NOT NULL DEFAULT 0`
-  - [ ] `chapter_number REAL NOT NULL DEFAULT -1`
-  - [ ] `scanlator VARCHAR(256)`
-  - [ ] `source_order INTEGER NOT NULL`
-  - [ ] `is_read BOOLEAN NOT NULL DEFAULT FALSE`
-  - [ ] `is_bookmarked BOOLEAN NOT NULL DEFAULT FALSE`
-  - [ ] `last_page_read INTEGER NOT NULL DEFAULT 0`
-  - [ ] `last_read_at BIGINT NOT NULL DEFAULT 0`
-  - [ ] `fetched_at BIGINT NOT NULL DEFAULT 0`
-  - [ ] `real_url VARCHAR(2048)`
-  - [ ] `is_downloaded BOOLEAN NOT NULL DEFAULT FALSE`
-  - [ ] `page_count INTEGER NOT NULL DEFAULT -1`
-  - [ ] `UNIQUE (manga_id, url)`
- [ ] `pages` table
-  - [ ] `id SERIAL PRIMARY KEY`
-  - [ ] `chapter_id INTEGER NOT NULL REFERENCES chapters(id) ON DELETE CASCADE`
-  - [ ] `"index" INTEGER NOT NULL`
-  - [ ] `url VARCHAR(2048) NOT NULL`
-  - [ ] `image_url TEXT`
- [ ] `source_meta` table
-  - [ ] `source_id BIGINT NOT NULL REFERENCES sources(id)`
-  - [ ] `key VARCHAR(256) NOT NULL`
-  - [ ] `value TEXT NOT NULL`
-  - [ ] `PRIMARY KEY (source_id, key)`
- [ ] Indexes
-  - [ ] `CREATE INDEX ON manga (source_id)`
-  - [ ] `CREATE INDEX ON manga (last_fetched_at)`
-  - [ ] `CREATE INDEX ON chapters (manga_id)`
-  - [ ] `CREATE INDEX ON pages (chapter_id)`
+- [x] `sources` table
+  - [x] `id BIGINT PRIMARY KEY`
+  - [x] `name VARCHAR(128) NOT NULL`
+  - [x] `lang VARCHAR(32) NOT NULL`
+  - [x] `is_nsfw BOOLEAN NOT NULL DEFAULT FALSE`
+- [x] `manga` table
+  - [x] `id SERIAL PRIMARY KEY`
+  - [x] `source_id BIGINT NOT NULL REFERENCES sources(id)`
+  - [x] `url VARCHAR(2048) NOT NULL`
+  - [x] `title VARCHAR(512) NOT NULL`
+  - [x] `initialized BOOLEAN NOT NULL DEFAULT FALSE`
+  - [x] `artist TEXT`, `author TEXT`, `description TEXT`, `genre TEXT`
+  - [x] `status INTEGER NOT NULL DEFAULT 0`
+  - [x] `thumbnail_url VARCHAR(2048)`
+  - [x] `thumbnail_last_fetched BIGINT NOT NULL DEFAULT 0`
+  - [x] `in_library BOOLEAN NOT NULL DEFAULT FALSE`
+  - [x] `in_library_at BIGINT NOT NULL DEFAULT 0`
+  - [x] `real_url VARCHAR(2048)`
+  - [x] `last_fetched_at BIGINT NOT NULL DEFAULT 0`
+  - [x] `chapters_last_fetched_at BIGINT NOT NULL DEFAULT 0`
+  - [x] `update_strategy VARCHAR(64) NOT NULL DEFAULT 'ALWAYS_UPDATE'`
+  - [x] `UNIQUE (source_id, url)`
+- [x] `chapters` table
+  - [x] `id SERIAL PRIMARY KEY`
+  - [x] `manga_id INTEGER NOT NULL REFERENCES manga(id) ON DELETE CASCADE`
+  - [x] `url VARCHAR(2048) NOT NULL`
+  - [x] `name VARCHAR(512) NOT NULL`
+  - [x] `date_upload BIGINT NOT NULL DEFAULT 0`
+  - [x] `chapter_number REAL NOT NULL DEFAULT -1`
+  - [x] `scanlator VARCHAR(256)`
+  - [x] `source_order INTEGER NOT NULL`
+  - [x] `is_read BOOLEAN NOT NULL DEFAULT FALSE`
+  - [x] `is_bookmarked BOOLEAN NOT NULL DEFAULT FALSE`
+  - [x] `last_page_read INTEGER NOT NULL DEFAULT 0`
+  - [x] `last_read_at BIGINT NOT NULL DEFAULT 0`
+  - [x] `fetched_at BIGINT NOT NULL DEFAULT 0`
+  - [x] `real_url VARCHAR(2048)`
+  - [x] `is_downloaded BOOLEAN NOT NULL DEFAULT FALSE`
+  - [x] `page_count INTEGER NOT NULL DEFAULT -1`
+  - [x] `UNIQUE (manga_id, url)`
+- [x] `pages` table
+  - [x] `id SERIAL PRIMARY KEY`
+  - [x] `chapter_id INTEGER NOT NULL REFERENCES chapters(id) ON DELETE CASCADE`
+  - [x] `"index" INTEGER NOT NULL`
+  - [x] `url VARCHAR(2048) NOT NULL`
+  - [x] `image_url TEXT`
+  - [x] `UNIQUE (chapter_id, "index")` — added for upsert support
+- [x] `source_meta` table
+  - [x] `source_id BIGINT NOT NULL REFERENCES sources(id)`
+  - [x] `key VARCHAR(256) NOT NULL`
+  - [x] `value TEXT NOT NULL`
+  - [x] `PRIMARY KEY (source_id, key)`
+- [x] Indexes
+  - [x] `CREATE INDEX ON manga (source_id)`
+  - [x] `CREATE INDEX ON manga (last_fetched_at)`
+  - [x] `CREATE INDEX ON chapters (manga_id)`
+  - [x] `CREATE INDEX ON pages (chapter_id)`

 ---

 ## 2.2 DB Initialization — `internal/db/db.go`

- [ ] `pgxpool.Pool` init from `DATABASE_URL` env var
-  - [ ] Configurable `MaxConns` (default 10)
-  - [ ] Configurable `MinConns` (default 2)
-  - [ ] Connection health check on startup
- [ ] Migration runner using `golang-migrate/migrate`
-  - [ ] Source: `iofs` (embed migration SQL files with `//go:embed`)
-  - [ ] Driver: `pgx5`
-  - [ ] Run `migrate.Up()` on startup; log version before/after
-  - [ ] Non-fatal on "no change" (already at latest)
- [ ] `Queries` struct wrapping sqlc-generated query clients
- [ ] `Close()` method to drain pool on shutdown
+- [x] `pgxpool.Pool` init from `DATABASE_URL` env var
+  - [x] Configurable `MaxConns` via `DB_MAX_CONNS` (default 10)
+  - [x] Configurable `MinConns` via `DB_MIN_CONNS` (default 2)
+  - [x] Connection health check on startup (`pool.Ping`)
+- [x] Migration runner using `golang-migrate/migrate`
+  - [x] Source: `iofs` (embed migration SQL files with `//go:embed`)
+  - [x] Driver: `pgx5` (DSN scheme rewritten from `postgres://` → `pgx5://`)
+  - [x] Run `migrate.Up()` on startup; log version before/after
+  - [x] Non-fatal on "no change" (`migrate.ErrNoChange`)
+- [x] `Queries` struct wrapping sqlc-generated query clients
+- [x] `Close()` method to drain pool on shutdown

 ---

 ## 2.3 SQL Queries — `internal/db/queries/`

-Write `.sql` files, then run `sqlc generate` to produce type-safe Go.
+Generated by `sqlc generate` (config: `sqlc.yaml`, `sql_package: pgx/v5`).

 ### `manga.sql`

- [ ] `UpsertManga` — INSERT ... ON CONFLICT (source_id, url) DO UPDATE all fields; returns row
- [ ] `GetMangaBySourceURL` — SELECT by source_id + url
- [ ] `GetMangaByID` — SELECT by primary key
- [ ] `ListMangaBySource` — SELECT by source_id ORDER BY last_fetched_at DESC
- [ ] `UpdateMangaDetails` — UPDATE artist/author/description/genre/status/thumbnail/initialized for a manga id
- [ ] `UpdateMangaFetchedAt` — UPDATE last_fetched_at = $now WHERE id = $id
- [ ] `UpdateChaptersFetchedAt` — UPDATE chapters_last_fetched_at WHERE id = $id
+- [x] `UpsertManga`
+- [x] `GetMangaBySourceURL`
+- [x] `GetMangaByID`
+- [x] `ListMangaBySource`
+- [x] `UpdateMangaDetails`
+- [x] `UpdateMangaFetchedAt`
+- [x] `UpdateChaptersFetchedAt`

 ### `chapter.sql`

- [ ] `UpsertChapter` — INSERT ... ON CONFLICT (manga_id, url) DO UPDATE name/date/chapter_number/scanlator/source_order
- [ ] `GetChapterByID`
- [ ] `ListChaptersByManga` — ORDER BY source_order
- [ ] `UpdateChapterFetchedAt`
+- [x] `UpsertChapter`
+- [x] `GetChapterByID`
+- [x] `ListChaptersByManga`
+- [x] `UpdateChapterFetchedAt`

 ### `page.sql`

- [ ] `UpsertPage` — INSERT ... ON CONFLICT (chapter_id, index) DO UPDATE url/image_url
- [ ] `ListPagesByChapter` — ORDER BY index
- [ ] `UpdatePageImageURL` — UPDATE image_url WHERE id = $id
+- [x] `UpsertPage`
+- [x] `ListPagesByChapter`
+- [x] `UpdatePageImageURL`

 ### `source.sql`

- [ ] `UpsertSource` — INSERT ... ON CONFLICT (id) DO UPDATE name/lang/is_nsfw
- [ ] `ListSources`
- [ ] `GetSourceByID`
- [ ] `GetSourceMeta` — SELECT value WHERE source_id = $id AND key = $key
- [ ] `SetSourceMeta` — INSERT ... ON CONFLICT DO UPDATE value
+- [x] `UpsertSource`
+- [x] `ListSources`
+- [x] `GetSourceByID`
+- [x] `GetSourceMeta`
+- [x] `SetSourceMeta`

 ---

 ## 2.4 sqlc Configuration — `sqlc.yaml`

- [ ] `version: "2"`
- [ ] engine: `postgresql`
- [ ] schema path pointing to `internal/db/migrations/`
- [ ] queries path pointing to `internal/db/queries/*.sql`
- [ ] output package: `internal/db/queries`
- [ ] `emit_json_tags: true`
- [ ] `emit_db_tags: true`
+- [x] `version: "2"`
+- [x] engine: `postgresql`
+- [x] schema path: `internal/db/migrations/`
+- [x] queries path: `internal/db/queries/*.sql`
+- [x] output package: `internal/db/queries`
+- [x] `sql_package: "pgx/v5"` — required for pgxpool compatibility
+- [x] `emit_json_tags: true`
+- [x] `emit_db_tags: true`
+- [x] `emit_pointers_for_null_types: true`

 ---

 ## 2.5 Data Flow & Cache Logic

-Four upsert flows — each called from the API handler before returning data.
+TTL env vars implemented in `internal/config/config.go`. Cache check logic lives in the API handler (Phase 5) using these values.

-### Manga List (`GetPopularManga` / `GetLatestUpdates` / `GetSearchManga`)
-
- [ ] Check `manga.last_fetched_at`: if within TTL (default 10 min) and `?refresh=false`, return DB rows
- [ ] Otherwise call source, then `UpsertManga` for each returned SManga
- [ ] Update `last_fetched_at` on upserted rows
- [ ] `UpsertSource` for the source record (idempotent on every list call)
-
-### Manga Detail (`GetMangaDetails`)
-
- [ ] Check `manga.initialized`: if true and within TTL (default 1h), return DB row
- [ ] Otherwise call source, then `UpdateMangaDetails` + set `initialized=true`
-
-### Chapter List (`GetChapterList`)
-
- [ ] Check `manga.chapters_last_fetched_at`: if within TTL (default 10 min), return DB rows
- [ ] Otherwise call source, then `UpsertChapter` for each SChapter (preserving `source_order` = slice index)
- [ ] Update `chapters_last_fetched_at` on the manga row
-
-### Page List (`GetPageList`)
-
- [ ] Check if pages exist for chapter in DB
- [ ] Otherwise call source, then `UpsertPage` for each Page
- [ ] If source needs a second call to resolve image URLs (via `GetImageURL`), call it and `UpdatePageImageURL`
- [ ] Store resolved `image_url` — do NOT cache for sources with signed/expiring URLs (globalcomix, etc.)
-
-### TTL Configuration
-
- [ ] `MANGA_LIST_TTL_SECONDS` env var (default 600)
- [ ] `MANGA_DETAIL_TTL_SECONDS` env var (default 3600)
- [ ] `CHAPTER_LIST_TTL_SECONDS` env var (default 600)
- [ ] `?refresh=true` query param bypasses TTL on any route
+- [x] `MANGA_LIST_TTL_SECONDS` env var (default 600)
+- [x] `MANGA_DETAIL_TTL_SECONDS` env var (default 3600)
+- [x] `CHAPTER_LIST_TTL_SECONDS` env var (default 600)
+- [ ] `?refresh=true` query param bypass — implemented in Phase 5 API handler
+- [ ] Per-flow cache check + upsert logic — implemented in Phase 5 API handler

 ---

 ## Checklist: Phase 2 Done When

- [ ] `golang-migrate` runs `001_init.sql` on a fresh DB without error
- [ ] `sqlc generate` completes without errors; generated files compile
- [ ] `UpsertManga` + `GetMangaBySourceURL` round-trip test passes
- [ ] `UpsertChapter` correctly sets `source_order` from slice position
- [ ] TTL cache logic returns DB rows on second call (verify no source HTTP call via log)
- [ ] `?refresh=true` bypasses TTL and re-fetches from source
- [ ] All tables visible in `psql` after API calls
+- [ ] `golang-migrate` runs `000001_init.up.sql` on a fresh DB without error (manual test with running postgres)
+- [x] `sqlc generate` completes without errors; generated files compile
+- [ ] `UpsertManga` + `GetMangaBySourceURL` round-trip test passes (requires running postgres)
+- [ ] `UpsertChapter` correctly sets `source_order` from slice position (requires running postgres)
+- [ ] TTL cache logic returns DB rows on second call (Phase 5)
+- [ ] `?refresh=true` bypasses TTL (Phase 5)
+- [ ] All tables visible in `psql` after API calls (Phase 5)