truf-server/openspec/changes/add-postman-source/tasks.md

1. Source Wiring

• 1.1 Add postman to CLI --source and --platform choices and source-to-platform mapping.
• 1.2 Add postman queue file support through the existing queue_files_for_args, prepare, and mark-checked flow.
• 1.3 Add postman to supervisor source configuration and dashboard source lists.
• 1.4 Add disabled-by-default sources.postman configuration with GitHub auth pool, search kinds, backfill/tail controls, cache path, and rate-limit settings.

2. Target Model And Cache

• 2.1 Define Postman target JSON formats for GitHub code search, npm package, PyPI package, local cache, and future URL targets.
• 2.2 Implement Postman target parsing and normalization in console_runner.py and scanner_db.py.
• 2.3 Implement durable Postman cache path resolution under the configured runtime directory.
• 2.4 Implement safe cache writes with SHA-256 content hashing, max artifact size checks, and origin metadata preservation.

3. GitHub Code Search Discovery

• 3.1 Implement GitHub code search queries for filename:postman_collection.json <query> and filename:postman_environment.json <query> based on search_kinds.
• 3.2 Implement bounded pagination using configured pages and per_page, respecting the GitHub 1000-result search cap.
• 3.3 Convert GitHub code search items into Postman target JSON containing repository, path, SHA, kind, API URL, and HTML URL.
• 3.4 Implement latest path commit lookup for max_file_age_days filtering.
• 3.5 Integrate existing known-page early stop behavior for Postman tail scans.

4. GitHub Token Pool And Rate Limits

• 4.1 Build a source-local GitHub token pool from configured auth_pool entries and fallback token settings.
• 4.2 Rotate tokens per GitHub code search, commit lookup, and content download request.
• 4.3 Mark only the failing token unavailable on primary rate limit, secondary rate limit, auth invalid, or auth forbidden responses.
• 4.4 Sleep until earliest known reset time, or configured fallback cooldown, when all GitHub tokens are unavailable.
• 4.5 Record token cooldown status in the source runtime state without exposing token values in logs or database snapshots.

5. Postman Artifact Scanning

• 5.1 Implement Postman content download from GitHub Contents API and cache it before scanning.
• 5.2 Implement scan_postman_target() to stage cached JSON in a temporary directory and run TruffleHog filesystem scanning.
• 5.3 Add Postman branch to scan_targets_batch() and pass timeout, detectors, excluded detectors, and verification flags.
• 5.4 Preserve nearby file context and apply existing noisy finding filters to Postman scan results.
• 5.5 Ensure Postman findings, errors, skipped reasons, and clean scans are persisted through existing JSONL and scanner database writes.

6. npm And PyPI Harvesting

• 6.1 Add a Postman artifact finder for extracted package directories that matches collection and environment filename patterns.
• 6.2 Cache npm package Postman artifacts before package temp directory cleanup and attach npm origin metadata.
• 6.3 Cache PyPI package Postman artifacts before package temp directory cleanup and attach PyPI origin metadata.
• 6.4 Enqueue harvested package artifacts into todo_postman.txt after package scan batches without failing the original package scan.
• 6.5 Deduplicate harvested package artifacts by content hash before enqueueing.

7. Postman-Aware Enrichment

• 7.1 Parse Postman collection and environment JSON into request, auth, header, query, body, and variable context maps.
• 7.2 Correlate TruffleHog finding locations or nearby context with Postman context maps.
• 7.3 Classify credential kind and provider using DetectorName, value shape, auth/header type, variable name, and endpoint host.
• 7.4 Detect common placeholders and assign placeholder or low-confidence classification.
• 7.5 Persist enrichment fields using existing finding enrichment/database columns where possible.

8. Observability And Configuration

• 8.1 Add Postman source cycle metrics, queue snapshots, target scan records, findings, and errors to existing database flows.
• 8.2 Add Postman queue counts to dashboard current queues and source health views.
• 8.3 Add redaction coverage for Postman/GitHub auth pool settings in config snapshots and logs.
• 8.4 Add backfill-friendly and tail-friendly config examples in config.yaml comments.

9. Verification

• 9.1 Run a small Postman GitHub discovery cycle with pages: 1, per_page: 10, and max_targets set.
• 9.2 Re-run the same cycle and verify duplicate targets are skipped through todo_postman.txt and checked_postman.txt.
• 9.3 Verify all-token rate-limit fallback with a simulated or controlled token-unavailable state.
• 9.4 Verify npm and PyPI harvesting using a package fixture containing collection and environment JSON files.
• 9.5 Verify database and dashboard visibility for Postman source cycles, queues, target scans, findings, and errors.
• 9.6 Run openspec status --change add-postman-source and ensure all implementation tasks are complete before archive.