goyomi/docs/phase2-database.md

Phase 2 — Database Layer

Persistent storage using PostgreSQL via pgx. Migrations run on startup. sqlc generates type-safe query code.

Reference schema modeled after Suwayomi-Server:

• /Users/achmad/Documents/Belajar/Web/Suwayomi-Server/server/src/main/kotlin/suwayomi/tachidesk/manga/model/table/
• Migrations: /Users/achmad/Documents/Belajar/Web/Suwayomi-Server/server/src/main/kotlin/suwayomi/tachidesk/server/database/migration/

---

2.1 Schema Migration — internal/db/migrations/001_init.sql

• sources table
  • id BIGINT PRIMARY KEY — generated via GenerateSourceID(name, lang) same as Tachiyomi
  • name VARCHAR(128) NOT NULL
  • lang VARCHAR(32) NOT NULL
  • is_nsfw BOOLEAN NOT NULL DEFAULT FALSE
• manga table
  • id SERIAL PRIMARY KEY
  • source_id BIGINT NOT NULL REFERENCES sources(id)
  • url VARCHAR(2048) NOT NULL
  • title VARCHAR(512) NOT NULL
  • initialized BOOLEAN NOT NULL DEFAULT FALSE
  • artist TEXT, author TEXT, description TEXT, genre TEXT
  • status INTEGER NOT NULL DEFAULT 0
  • thumbnail_url VARCHAR(2048)
  • thumbnail_last_fetched BIGINT NOT NULL DEFAULT 0
  • in_library BOOLEAN NOT NULL DEFAULT FALSE
  • in_library_at BIGINT NOT NULL DEFAULT 0
  • real_url VARCHAR(2048)
  • last_fetched_at BIGINT NOT NULL DEFAULT 0
  • chapters_last_fetched_at BIGINT NOT NULL DEFAULT 0
  • update_strategy VARCHAR(64) NOT NULL DEFAULT 'ALWAYS_UPDATE'
  • UNIQUE (source_id, url)
• chapters table
  • id SERIAL PRIMARY KEY
  • manga_id INTEGER NOT NULL REFERENCES manga(id) ON DELETE CASCADE
  • url VARCHAR(2048) NOT NULL
  • name VARCHAR(512) NOT NULL
  • date_upload BIGINT NOT NULL DEFAULT 0
  • chapter_number REAL NOT NULL DEFAULT -1
  • scanlator VARCHAR(256)
  • source_order INTEGER NOT NULL
  • is_read BOOLEAN NOT NULL DEFAULT FALSE
  • is_bookmarked BOOLEAN NOT NULL DEFAULT FALSE
  • last_page_read INTEGER NOT NULL DEFAULT 0
  • last_read_at BIGINT NOT NULL DEFAULT 0
  • fetched_at BIGINT NOT NULL DEFAULT 0
  • real_url VARCHAR(2048)
  • is_downloaded BOOLEAN NOT NULL DEFAULT FALSE
  • page_count INTEGER NOT NULL DEFAULT -1
  • UNIQUE (manga_id, url)
• pages table
  • id SERIAL PRIMARY KEY
  • chapter_id INTEGER NOT NULL REFERENCES chapters(id) ON DELETE CASCADE
  • "index" INTEGER NOT NULL
  • url VARCHAR(2048) NOT NULL
  • image_url TEXT
• source_meta table
  • source_id BIGINT NOT NULL REFERENCES sources(id)
  • key VARCHAR(256) NOT NULL
  • value TEXT NOT NULL
  • PRIMARY KEY (source_id, key)
• Indexes
  • CREATE INDEX ON manga (source_id)
  • CREATE INDEX ON manga (last_fetched_at)
  • CREATE INDEX ON chapters (manga_id)
  • CREATE INDEX ON pages (chapter_id)

---

2.2 DB Initialization — internal/db/db.go

• pgxpool.Pool init from DATABASE_URL env var
  • Configurable MaxConns (default 10)
  • Configurable MinConns (default 2)
  • Connection health check on startup
• Migration runner using golang-migrate/migrate
  • Source: iofs (embed migration SQL files with //go:embed)
  • Driver: pgx5
  • Run migrate.Up() on startup; log version before/after
  • Non-fatal on "no change" (already at latest)
• Queries struct wrapping sqlc-generated query clients
• Close() method to drain pool on shutdown

---

2.3 SQL Queries — internal/db/queries/

Write .sql files, then run sqlc generate to produce type-safe Go.

manga.sql

• UpsertManga — INSERT ... ON CONFLICT (source_id, url) DO UPDATE all fields; returns row
• GetMangaBySourceURL — SELECT by source_id + url
• GetMangaByID — SELECT by primary key
• ListMangaBySource — SELECT by source_id ORDER BY last_fetched_at DESC
• UpdateMangaDetails — UPDATE artist/author/description/genre/status/thumbnail/initialized for a manga id
• UpdateMangaFetchedAt — UPDATE last_fetched_at = $now WHERE id = $id
• UpdateChaptersFetchedAt — UPDATE chapters_last_fetched_at WHERE id = $id

chapter.sql

• UpsertChapter — INSERT ... ON CONFLICT (manga_id, url) DO UPDATE name/date/chapter_number/scanlator/source_order
• GetChapterByID
• ListChaptersByManga — ORDER BY source_order
• UpdateChapterFetchedAt

page.sql

• UpsertPage — INSERT ... ON CONFLICT (chapter_id, index) DO UPDATE url/image_url
• ListPagesByChapter — ORDER BY index
• UpdatePageImageURL — UPDATE image_url WHERE id = $id

source.sql

• UpsertSource — INSERT ... ON CONFLICT (id) DO UPDATE name/lang/is_nsfw
• ListSources
• GetSourceByID
• GetSourceMeta — SELECT value WHERE source_id = $id AND key = $key
• SetSourceMeta — INSERT ... ON CONFLICT DO UPDATE value

---

2.4 sqlc Configuration — sqlc.yaml

• version: "2"
• engine: postgresql
• schema path pointing to internal/db/migrations/
• queries path pointing to internal/db/queries/*.sql
• output package: internal/db/queries
• emit_json_tags: true
• emit_db_tags: true

---

2.5 Data Flow & Cache Logic

Four upsert flows — each called from the API handler before returning data.

Manga List (GetPopularManga / GetLatestUpdates / GetSearchManga)

• Check manga.last_fetched_at: if within TTL (default 10 min) and ?refresh=false, return DB rows
• Otherwise call source, then UpsertManga for each returned SManga
• Update last_fetched_at on upserted rows
• UpsertSource for the source record (idempotent on every list call)

Manga Detail (GetMangaDetails)

• Check manga.initialized: if true and within TTL (default 1h), return DB row
• Otherwise call source, then UpdateMangaDetails + set initialized=true

Chapter List (GetChapterList)

• Check manga.chapters_last_fetched_at: if within TTL (default 10 min), return DB rows
• Otherwise call source, then UpsertChapter for each SChapter (preserving source_order = slice index)
• Update chapters_last_fetched_at on the manga row

Page List (GetPageList)

• Check if pages exist for chapter in DB
• Otherwise call source, then UpsertPage for each Page
• If source needs a second call to resolve image URLs (via GetImageURL), call it and UpdatePageImageURL
• Store resolved image_url — do NOT cache for sources with signed/expiring URLs (globalcomix, etc.)

TTL Configuration

• MANGA_LIST_TTL_SECONDS env var (default 600)
• MANGA_DETAIL_TTL_SECONDS env var (default 3600)
• CHAPTER_LIST_TTL_SECONDS env var (default 600)
• ?refresh=true query param bypasses TTL on any route

---

Checklist: Phase 2 Done When

• golang-migrate runs 001_init.sql on a fresh DB without error
• sqlc generate completes without errors; generated files compile
• UpsertManga + GetMangaBySourceURL round-trip test passes
• UpsertChapter correctly sets source_order from slice position
• TTL cache logic returns DB rows on second call (verify no source HTTP call via log)
• ?refresh=true bypasses TTL and re-fetches from source
• All tables visible in psql after API calls