tanabata

Tanabata File Manager — Go Project Structure

Backend counterpart of ARCHITECTURE.md. This document details the Go layout, the layer rules, and the key backend decisions.

Stack

• Router: Gin
• Database: pgx v5 (pgxpool)
• Migrations: goose v3 + go:embed (auto-applied on startup)
• Auth: JWT (golang-jwt/jwt/v5), Bearer access tokens + rotating refresh tokens
• Config: environment variables via .env (joho/godotenv)
• Logging: slog (stdlib)
• Metadata: exiftool (external, preferred) with a pure-Go EXIF fallback (rwcarlsen/goexif)
• Thumbnails / previews: vipsthumbnail (external, shrink-on-load) and ffmpeg (video frames), with a pure-Go fallback (disintegration/imaging)
• Near-duplicate detection: 64-bit dHash perceptual hashing + a BK-tree / Hamming-distance pairing (internal/imagehash, internal/service/duplicate_*)
• Architecture: Clean Architecture (domain → service → repository/handler)

The binary is fully static (CGO_ENABLED=0). External tools are invoked as subprocesses when present and are optional — the pure-Go paths keep the server working without them.

Monorepo Layout

tanabata/
├── backend/                            ← Go project
├── frontend/                           ← SvelteKit project
├── openapi.yaml                        ← Shared API contract
├── docker-compose.yml
├── Dockerfile
├── .env.example
└── README.md

Backend Directory Layout

backend/
├── cmd/
│   ├── server/
│   │   └── main.go                     # Entrypoint: config → DB → migrate → bootstrap admin → wire → serve
│   └── dedup/
│       └── main.go                     # Offline maintenance CLI: perceptual-hash backfill + duplicate-pairs rescan
│
├── internal/
│   │
│   ├── domain/                         # Pure business entities & value objects (stdlib only)
│   │   ├── file.go                     # File, FileFilter, FileListParams, FilePage
│   │   ├── tag.go                      # Tag, TagRule
│   │   ├── category.go                 # Category
│   │   ├── pool.go                     # Pool, PoolFile
│   │   ├── user.go                     # User, Session
│   │   ├── acl.go                      # Permission, ObjectType
│   │   ├── audit.go                    # AuditEntry, ActionType
│   │   ├── duplicate.go                # DuplicatePair, PHashEntry
│   │   ├── context.go                  # WithUser / UserFromContext (identity + session in ctx)
│   │   └── errors.go                   # Domain error types (ErrNotFound, ErrForbidden, …)
│   │
│   ├── port/                           # Interfaces (ports) — contracts between layers
│   │   ├── repository.go               # Transactor, FileRepo, TagRepo, TagRuleRepo, CategoryRepo,
│   │   │                               # PoolRepo, UserRepo, SessionRepo, ACLRepo, AuditRepo,
│   │   │                               # MimeRepo, DuplicatePairRepo, DismissalRepo
│   │   └── storage.go                  # FileStorage (originals + thumbnail/preview cache)
│   │
│   ├── service/                        # Business logic (use cases)
│   │   ├── file_service.go             # Upload, update, delete, trash/restore, replace, import, filter/list
│   │   ├── tag_service.go              # CRUD + auto-tag (rule) application
│   │   ├── category_service.go         # CRUD (thin: repo + ACL + audit)
│   │   ├── pool_service.go             # CRUD + file ordering, add/remove files
│   │   ├── auth_service.go             # Login, logout, JWT issue/refresh, content tokens, sessions
│   │   ├── acl_service.go              # Permission checks, grant/revoke
│   │   ├── audit_service.go            # Log actions, query audit log
│   │   ├── user_service.go             # Profile update, admin CRUD, block/unblock, EnsureAdmin
│   │   ├── duplicate_service.go        # Cluster / resolve (merge) / dismiss + rescan orchestration
│   │   ├── duplicate_index.go          # BK-tree, Hamming pairing, connected-component clustering
│   │   └── metadata.go                 # EXIF / media metadata extraction (exiftool + pure-Go fallback)
│   │
│   ├── handler/                        # HTTP layer (Gin handlers)
│   │   ├── router.go                   # Route registration, middleware, security headers, SPA fallback
│   │   ├── middleware.go               # Auth middleware (JWT / content token → context)
│   │   ├── ratelimit.go                # Per-IP token-bucket limiter for /auth
│   │   ├── response.go                 # Error/success builders, domain-error → HTTP mapping
│   │   ├── static.go                   # Built SPA serving + index.html fallback
│   │   ├── file_handler.go             # /files endpoints
│   │   ├── duplicate_handler.go        # /files/duplicates endpoints
│   │   ├── tag_handler.go              # /tags endpoints (+ file–tag relations)
│   │   ├── category_handler.go         # /categories endpoints
│   │   ├── pool_handler.go             # /pools endpoints
│   │   ├── auth_handler.go             # /auth endpoints
│   │   ├── acl_handler.go              # /acl endpoints
│   │   ├── user_handler.go             # /users endpoints
│   │   └── audit_handler.go            # /audit endpoint
│   │
│   ├── db/                             # Database adapters
│   │   ├── db.go                       # Shared helpers: Querier, tx-from-context, ScanRow, limit/offset clamps
│   │   └── postgres/                   # PostgreSQL implementation
│   │       ├── postgres.go             # pgxpool init, Transactor, conn-or-tx helper
│   │       ├── file_repo.go            # FileRepo (incl. perceptual-hash projections)
│   │       ├── tag_repo.go             # TagRepo + TagRuleRepo
│   │       ├── category_repo.go        # CategoryRepo
│   │       ├── pool_repo.go            # PoolRepo
│   │       ├── user_repo.go            # UserRepo
│   │       ├── session_repo.go         # SessionRepo
│   │       ├── acl_repo.go             # ACLRepo
│   │       ├── audit_repo.go           # AuditRepo
│   │       ├── mime_repo.go            # MimeRepo
│   │       ├── duplicate_repo.go       # DuplicatePairRepo + DismissalRepo
│   │       └── filter_parser.go        # Filter DSL → SQL WHERE clause builder
│   │
│   ├── storage/                        # File storage adapter
│   │   └── disk.go                     # FileStorage on disk: originals + thumbnail/preview cache
│   │                                   # (vipsthumbnail / ffmpeg / pure-Go imaging)
│   │
│   ├── imagehash/                      # Perceptual hashing (64-bit dHash) for near-duplicate detection
│   │   └── imagehash.go
│   │
│   ├── integration/                    # End-to-end HTTP tests against a disposable Postgres
│   │   └── server_test.go
│   │
│   └── config/                         # Configuration
│       └── config.go                   # Config struct + loader from env vars
│
├── migrations/                         # SQL migration files (goose format), embedded via go:embed
│   ├── 001_init_schemas.sql
│   ├── 002_core_tables.sql
│   ├── 003_data_tables.sql
│   ├── 004_acl_tables.sql
│   ├── 005_activity_tables.sql
│   ├── 006_indexes.sql
│   ├── 007_seed_data.sql
│   └── embed.go                        # //go:embed *.sql → migrations.FS
│
├── go.mod
└── go.sum

Layer Dependency Rules

handler  →  service  →  port (interfaces)  ←  db/postgres / storage
                ↓
              domain (entities, value objects, errors)

• domain/: zero imports from other internal packages. Only stdlib.
• port/: imports only domain/. Defines interfaces.
• service/: imports domain/ and port/. Never imports db/ or handler/.
• handler/: imports domain/ and service/. Never imports db/.
• db/postgres/: imports domain/, port/, and db/ (common helpers). Implements port interfaces.
• db/: imports domain/ and port/. Shared utilities for all DB adapters.
• storage/: imports domain/ and port/. Implements FileStorage.
• imagehash/: leaf package (stdlib + image libs); used by service/ and storage/.

No layer may import a layer above it. No circular dependencies.

Key Design Decisions

Dependency Injection (Wiring)

Manual wiring in cmd/server/main.go. No DI frameworks. Constructors take their collaborators explicitly; the shape below matches the real signatures.

// Pseudocode — see cmd/server/main.go for the exact calls.
pool := postgres.NewPool(ctx, cfg.DatabaseURL)
goose.Up(stdlib.OpenDBFromPool(pool), ".")   // migrations.FS embedded

// Storage
diskStorage := storage.NewDiskStorage(
    cfg.FilesPath, cfg.ThumbsCachePath,
    cfg.ThumbWidth, cfg.ThumbHeight, cfg.PreviewWidth, cfg.PreviewHeight,
    cfg.ThumbMaxPixels, cfg.ThumbConcurrency,
)

// Repos (all from internal/db/postgres/)
fileRepo := postgres.NewFileRepo(pool)
// … tag, tagRule, category, pool, user, session, acl, audit, mime,
//    duplicatePair, dismissal repos + transactor

// Services
authSvc := service.NewAuthService(userRepo, sessionRepo,
    cfg.JWTSecret, cfg.JWTAccessTTL, cfg.JWTRefreshTTL, cfg.ContentTokenTTL)
aclSvc  := service.NewACLService(aclRepo, fileRepo, tagRepo, categoryRepo, poolRepo, transactor)
auditSvc := service.NewAuditService(auditRepo)
tagSvc  := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc, transactor)
dupSvc  := service.NewDuplicateService(fileRepo, duplicatePairRepo, dismissalRepo,
    aclSvc, auditSvc, transactor, cfg.DuplicateHashThreshold)
fileSvc := service.NewFileService(fileRepo, mimeRepo, diskStorage,
    aclSvc, auditSvc, tagSvc, transactor, cfg.ImportPath)
// … category, pool, user services

// Bootstrap the initial admin from env (idempotent).
userSvc.EnsureAdmin(ctx, cfg.AdminUsername, cfg.AdminPassword)

// Handlers → router (also wires trusted proxies + optional static SPA dir)
router, _ := handler.NewRouter(authMiddleware, authHandler, fileHandler,
    duplicateHandler, tagHandler, categoryHandler, poolHandler,
    userHandler, aclHandler, auditHandler, cfg.StaticDir, cfg.TrustedProxies)
srv.ListenAndServe()

Context Propagation

Every service method receives context.Context as the first argument. The auth middleware parses the JWT and puts the caller’s identity (user id, admin flag, session id) into the context. Services read it for ACL checks and audit logging.

// handler/middleware.go
claims := parseJWT(c.GetHeader("Authorization"))
ctx := domain.WithUser(c.Request.Context(), claims.UserID, claims.IsAdmin, claims.SessionID)
c.Request = c.Request.WithContext(ctx)

// domain/context.go
func WithUser(ctx context.Context, userID int16, isAdmin bool, sessionID int) context.Context
func UserFromContext(ctx context.Context) (userID int16, isAdmin bool, sessionID int)

Transaction Management

The Transactor port lets services compose multiple repo calls atomically. The postgres implementation stores the active pgx.Tx in the context; repo methods pick it up via a conn-or-tx helper, so the same method works inside or outside a transaction.

// port/repository.go
type Transactor interface {
    WithTx(ctx context.Context, fn func(ctx context.Context) error) error
}

// service/file_service.go (sketch)
func (s *FileService) Upload(ctx context.Context, p UploadParams) (*domain.File, error) {
    return s.tx.WithTx(ctx, func(ctx context.Context) error {
        created, err := s.files.Create(ctx, f)   // uses tx from ctx
        // apply initial tags, etc., in the same tx
        return err
    })
}

ACL Check Pattern

ACL logic is centralized in ACLService. Other services call it before any mutation or retrieval. The model is private-by-default: admins see everything; otherwise a public flag, creator ownership, or an explicit acl.permissions grant is required.

// service/acl_service.go (shape)
func (s *ACLService) CanView(ctx context.Context, userID int16, isAdmin bool,
    creatorID int16, isPublic bool, objectType int16, objectID uuid.UUID) (bool, error)
func (s *ACLService) CanEdit(ctx context.Context, userID int16, isAdmin bool,
    creatorID int16, objectType int16, objectID uuid.UUID) (bool, error)

Error Mapping

Domain errors → HTTP status codes (handled in handler/response.go):

Domain Error	HTTP Status	Error Code
ErrNotFound	404	not_found
ErrForbidden	403	forbidden
ErrUnauthorized	401	unauthorized
ErrConflict	409	conflict
ErrValidation	400	validation_error
ErrUnsupportedMIME	415	unsupported_mime
(unexpected)	500	internal_error

Filter DSL

The DSL parser lives in db/postgres/filter_parser.go because it produces SQL WHERE clauses — a PostgreSQL-specific adapter concern. The service layer passes the raw DSL string down; the repository parses it and builds the query. For a different DBMS, a corresponding parser would live in db/<dbms>/filter_parser.go.

// domain/file.go
type FileListParams struct {
    Filter    string    // raw DSL string
    Sort      string
    Order     string
    Cursor    string
    Anchor    *uuid.UUID
    Direction string    // "forward" or "backward"
    Limit     int
    Trash     bool
    Search    string
}

The DSL grammar itself is documented in openapi.yaml (the filter query parameter), so the contract stays in one place.

JWT Structure

type Claims struct {
    jwt.RegisteredClaims
    UserID    int16 `json:"uid"`
    IsAdmin   bool  `json:"adm"`
    SessionID int   `json:"sid"`
}

Access token: short-lived (15 min default). Refresh token: long-lived (30 days default), rotated on use, stored as a hash in activity.sessions. A separate content token (default 6 h) is a single-file capability minted for media URLs, so a long video keeps streaming past access-token expiry — see CONTENT_TOKEN_TTL in .env.example.

Perceptual Duplicate Detection

Images are dHash-ed inline on upload (internal/imagehash); video hashes are backfilled by the dedup CLI (ffmpeg stays off the upload path). A rescan rebuilds data.duplicate_pairs by inserting every pair within DUPLICATE_HASH_THRESHOLD Hamming distance (BK-tree lookups, not O(N²)); the duplicates API then groups pairs into connected-component clusters. See service/duplicate_service.go and service/duplicate_index.go.

Configuration (.env)

Every variable the server reads is documented in .env.example (1:1 with config.Config). Required at startup: JWT_SECRET, ADMIN_PASSWORD, DATABASE_URL, FILES_PATH, THUMBS_CACHE_PATH, IMPORT_PATH. Everything else has a sensible default (see config.go).

This site is open source. Improve this page.