# Changelog of Twitter Scraper (`thisismegant/twitter-scraper`) Actor - **URL**: https://apify.com/thisismegant/twitter-scraper/changelog.md - **Full Actor documentation**: https://apify.com/thisismegant/twitter-scraper.md ### 2026-07-05 #### Added - Created `.actor/output_schema.json` and linked it via `"output": "./output_schema.json"` in `actor.json` to resolve the Apify Console "Output schema" missing error. - Added `storages.dataset.views` to `.actor/actor.json` to define an Output Schema (Overview Table) in the Apify Console. This fixes the "Output schema" missing error and presents a clean UI for datasets. #### Changed - None. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - Next recommended tasks: Implement Automated Pagination to fetch subsequent batches of tweets. #### Added - Added fields to the Retweet output schema (`twitterRetweetOutputSchema`) to capture the original retweeted content (author username, text, and engagement metrics). #### Changed - Updated `src/parsers/twitter-graphql-parser.ts` to extract the full original tweet context from the internal `retweeted_status_result` property. - Updated `tests/twitter-output-schema.test.ts` to expect original tweet properties for Retweets. #### Fixed - Fixed an issue where retweets only contained the raw `tweetId` and timestamps without any readable content. #### Refactored - None. #### Notes - Breaking changes: None. Retweets now have more contextual data attached. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: None. - Next recommended session: Implement Automated Pagination to fetch subsequent batches of tweets. ### 2026-07-05 #### Added - None. #### Changed - Updated `src/parsers/twitter-graphql-parser.ts` to fallback `authorUsername` to the scraped `identifier` if Twitter's GraphQL omits the `core.user_results.result.legacy.screen_name` property. #### Fixed - Fixed an issue where `authorUsername` would be marked as "unknown" for certain timeline tweets. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: None. - Next recommended session: Implement Automated Pagination to fetch subsequent batches of tweets. ### 2026-07-05 #### Added - None. #### Changed - Removed the string prefixes (`profile:`, `tweet:`, `reply:`, `retweet:`) from the generated `id` fields in parsed output records. IDs are now purely the numeric Twitter ID. - Updated `tests/twitter-output-schema.test.ts` to expect plain IDs. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: Output record IDs no longer have a type prefix. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: Paginating Twitter GraphQL timeline is not yet implemented. - Next recommended session: Implement Automated Pagination to fetch subsequent batches of tweets. ### 2026-07-05 #### Added - Added `src/parsers/twitter-graphql-parser.ts` to recursively parse and extract normalized `tweet`, `reply`, and `retweet` records from raw Twitter GraphQL JSON payloads. - Added a fast-path in `src/apify/index.ts` for profile scraping that directly maps data from the `api.fxtwitter.com` API without executing Twitter GraphQL requests, vastly improving performance and avoiding rate limits. #### Changed - Updated `src/utils/user-resolver.ts` to export a `resolveUsernameToProfile` function that returns a full `ParsedTwitterOutputRecord` for a profile. - Updated `src/apify/index.ts` to implement the data parsing engine and conditionally route to the profile fast-path or the GraphQL execution path based on the user's `scrapeType`. - Modified `tests/apify.test.ts` to validate the new fast-path and filtered data output expectations. #### Fixed - Fixed various TypeScript strictly-typed assignment errors (`any`) within the parser and string assignments. - Fixed testing errors related to `Actor.pushData` no longer blindly pushing the raw HTTP response, but instead pushing the filtered & normalized records. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: Paginating Twitter GraphQL timeline is not yet implemented (we only fetch the first 20-100 tweets). - Next recommended session: Implement Automated Pagination to fetch subsequent batches of tweets. ### 2026-07-05 #### Added - Added `.actor/README.md` to serve as the Apify Storefront documentation, complete with step-by-step instructions on how to use the cURL scraper. - Added `README.md` in the project root to serve as the Developer documentation, outlining the architecture, tech stack, and local development setup. #### Changed - Deleted the obsolete `docs/` folder which contained legacy CLI design documents. - Deleted numerous empty/obsolete `README.md` files scattered throughout the `src/` and `tests/` directories. #### Fixed - Removed `curlCommand` from the Apify Input Schema so end-users no longer need to provide cookies. - Refactored `src/apify/index.ts` to read cookies securely from Apify Environment Variables (`process.env.TWITTER_CURL_COMMAND`), with a fallback to `fixtures/curl-example.txt` for local development. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: None. - Next recommended session: Implement automated Pagination or Data Parsing to refine the raw GraphQL JSON. ### 2026-07-05 #### Added - Transformed the Apify Actor entrypoint (`src/apify/index.ts`) from a Developer CLI wrapper into a true End-User UI processor. - Added extraction of User inputs for `scrapeType`, `identifier`, and `curlCommand`. - Apify runner now automatically converts user inputs into a structured `JobDefinition` and stores it into the Apify Key-Value store alongside extracted `curlCommand` headers. - Implemented the HTTP Execution Engine in the Apify Actor using Axios (`src/services/axios-http-client.ts`). - Successfully executes the parsed cURL configuration to fetch data directly from Twitter's GraphQL APIs. - Saves the raw Twitter JSON response directly into the Apify Dataset (via `Actor.pushData`) and Key-Value store. #### Changed - Completely redesigned `.actor/input_schema.json` to feature an End-User interface, with target dropdowns (`profile`, `tweet`, etc.), target identifiers, max items, and a textarea for raw Network API cURL string. - Refactored `tests/apify.test.ts` to validate the new end-user input flow instead of CLI routing. - Files modified: `.actor/input_schema.json`, `src/apify/index.ts`, `tests/apify.test.ts`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Deleted - Removed `src/cli/`, `src/runtime/`, and associated tests (CLI is obsolete). - Removed `src/browser/` and `fixtures/network-api/` (Puppeteer interception is obsolete). - Removed `src/storage/` (Replaced by native Apify storage). - Removed obsolete service files (`src/services/job-execution-gates.ts`, etc.) and `src/index.ts`. - Removed CLI commands from `package.json`. #### Notes - Breaking changes: Apify Actor no longer accepts `command` or routes to the local CLI. It only accepts `scrapeType`, `identifier`, and `curlCommand`. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: The actual scraping execution/collector is not implemented yet; it only extracts configuration. - Next recommended session: Implement the HTTP Request Engine (Axios/Got) to fetch the Twitter GraphQL APIs using the parsed configuration. ### 2026-07-05 #### Added - None. #### Changed - Updated `fixtures/curl-example.txt` with a real Twitter GraphQL Network API cURL example for testing. #### Fixed - Fixed `curl-parser.ts` to explicitly capture `-b`/`--cookie` and `-A`/`--user-agent` values into the headers object instead of skipping them. - Removed redundant nullish coalescing operators (`?? ''`) in `src/utils/curl-parser.ts` that were causing ESLint errors (`@typescript-eslint/no-unnecessary-condition`). - Files modified: `src/utils/curl-parser.ts`, `tests/curl-parser.test.ts`, `CHANGELOG.md`. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: None. - Next recommended session: Continue with the next planned feature for the Twitter/X scraper or Apify deployment. ### 2026-07-05 #### Added - Added Apify Actor configuration and input schema to deploy the foundation CLI and commands directly onto Apify. - Added programmatic Apify Actor runner integration that maps actor input payloads, manages temporary file serialization, executes CLI tasks, logs command outputs, and writes results to the Apify Key-Value Store. - Added comprehensive unit tests for the Apify Actor entrypoint. - Files created: `.actor/actor.json`, `.actor/input_schema.json`, `src/apify/index.ts`, `tests/apify.test.ts`. #### Changed - Updated `package.json` to include `apify` dependency and `start` script mapping to compiled Apify entrypoint. - Updated `Dockerfile` default execution target to start the Apify Actor runner. - Files modified: `package.json`, `Dockerfile`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: Run `npm install` locally to install the newly added `apify` SDK. - New dependencies: `apify` (v3.2.6). - Removed dependencies: None. - New environment variables: None. - New npm scripts: `start` to run the compiled Apify Actor entrypoint. - Known limitations: Apify runner is mapped to foundation CLI commands; actual Twitter/X scraping logic, browser automation, and data parser pipelines remain out of scope for this foundation phase. - Next recommended session: add storage repository saving integrations or cURL parser support. ### 2026-07-05 #### Added - Added no-op browser boundary adapter to keep runtime composition browser-neutral. - Files created: `src/browser/noop-browser.ts`. #### Changed - Removed Playwright-owned runtime paths, capture command, browser adapter, browser lifecycle documentation, dependency, and related tests. - Updated runtime composition to use a no-op browser boundary. - Ignored root `auth/` so locally copied browser/session material is never committed. - Updated README, CLI documentation, Network API capture documentation, scraping design notes, browser folder documentation, package metadata, CLI tests, infrastructure tests, and task tracking. - Files modified: `.gitignore`, `package.json`, `package-lock.json`, `src/runtime/app.ts`, `src/cli/app.ts`, `src/browser/README.md`, `src/browser/network-api-capture-listener.ts`, `tests/cli.test.ts`, `tests/infrastructure.test.ts`, `README.md`, `docs/cli.md`, `docs/network-api-capture.md`, `docs/twitter-x-scraping-design.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - Removed uncommitted Playwright capture flow that conflicted with manual-browser credential handling. - Made storage CLI tests independent from the local `data/` directory state. #### Refactored - Replaced Playwright browser runtime adapter with a no-op browser boundary. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: `playwright`. - New environment variables: None. - New npm scripts: None. - Known limitations: browser automation is no longer owned by the project; manual browser Network requests must be copied in by the user; no HTML scraping, DOM parsing, parser implementation, collector implementation, endpoint catalog, or direct axios calls to Twitter/X have been added. - Next recommended session: add cURL import support for manually copied browser Network requests. ### 2026-07-05 #### Added - Added generic Network API capture listener skeleton. - Added attach/detach support for Playwright-like response events. - Added listener tests for matching JSON capture, document-response rejection, and detach behavior. - Files created: `src/browser/network-api-capture-listener.ts`, `tests/network-api-capture-listener.test.ts`. #### Changed - Updated Network API capture documentation and browser folder documentation. - Aligned Network API snapshot optional fields with strict Zod optional inference. - Files modified: `src/browser/network-api-capture.ts`, `docs/network-api-capture.md`, `src/browser/README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: listener is generic and not wired into runtime execution; no Twitter/X browser navigation, login, parser, collector, endpoint catalog, direct Twitter/X API call, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: add a local non-Twitter harness for exercising the listener, or define storage repository conventions for normalized records. ### 2026-07-05 #### Added - Added normalized output fixtures. - Added `outputs:validate` CLI command for local normalized output validation. - Added fixture tests for normalized output examples. - Added CLI tests for valid and invalid output validation. - Files created: `fixtures/output/profile-output.json`, `fixtures/output/tweet-output.json`, `src/cli/commands/output-validate-command.ts`. #### Changed - Registered `outputs:validate` in the default CLI registry, command list, and help output. - Updated README, CLI documentation, fixture documentation, output schema documentation, fixture tests, CLI tests, and task tracking. - Files modified: `src/cli/app.ts`, `tests/fixtures.test.ts`, `tests/cli.test.ts`, `README.md`, `docs/cli.md`, `docs/fixtures.md`, `docs/output-schemas.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: output validation reads local fixture files only and does not write records; no parser, collector, job execution path, runtime network listener, direct Twitter/X API call, endpoint catalog, browser navigation, login, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: define storage repository conventions for normalized records. ### 2026-07-05 #### Added - Added normalized Twitter/X output record TypeScript contracts. - Added Zod schemas for profile, tweet, reply, and retweet output records. - Added output schema tests. - Added output schema documentation. - Files created: `src/models/twitter-output.ts`, `src/models/twitter-output.schema.ts`, `docs/output-schemas.md`, `tests/twitter-output-schema.test.ts`. #### Changed - Updated README, docs index, models documentation, and task tracking. - Files modified: `README.md`, `docs/README.md`, `src/models/README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: output schemas are validation-only contracts; no parser, collector, job execution path, runtime network listener, direct Twitter/X API call, endpoint catalog, browser navigation, login, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: add validation-only output fixtures and an output fixture validation command, or define storage repository conventions for normalized records. ### 2026-07-04 #### Added - Added local fixture conventions documentation. - Added local job definition fixtures. - Added local Network API capture rule fixtures with placeholder URL patterns. - Added fixture validation tests. - Files created: `fixtures/README.md`, `fixtures/jobs/profile-job.json`, `fixtures/jobs/tweet-job.json`, `fixtures/network-api/profile-rules.json`, `fixtures/network-api/tweet-rules.json`, `docs/fixtures.md`, `tests/fixtures.test.ts`. #### Changed - Updated README with fixture validation command examples. - Updated job and Network API documentation next-session guidance. - Updated docs index and task tracking. - Files modified: `README.md`, `docs/job-contracts.md`, `docs/network-api-capture.md`, `docs/README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: fixtures are placeholder-only validation examples and not endpoint catalogs; no job execution path, runtime network listener, direct Twitter/X API call, endpoint catalog, browser navigation, login, collector implementation, parser implementation, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: define storage output schema contracts or add fixture validation automation. ### 2026-07-04 #### Added - Added execution approval gate evaluation service. - Added `jobs:gates` CLI command for pending, non-executable gate checks. - Added execution gate service tests. - Added CLI tests for valid and invalid gate checks. - Files created: `src/services/job-execution-gates.ts`, `src/cli/commands/job-gates-command.ts`, `tests/job-execution-gates.test.ts`. #### Changed - Registered `jobs:gates` in the default CLI registry, command list, and help output. - Updated README, CLI documentation, job contract documentation, services documentation, models documentation, and task tracking. - Files modified: `src/cli/app.ts`, `src/services/README.md`, `src/models/README.md`, `tests/cli.test.ts`, `docs/cli.md`, `docs/job-contracts.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: gate evaluation is informational only; no job execution path, runtime network listener, direct Twitter/X API call, endpoint catalog, browser navigation, login, collector implementation, parser implementation, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: define fixture conventions for local job and Network API capture rule examples. ### 2026-07-04 #### Added - Added dry-run job orchestration planner. - Added `jobs:plan` CLI command for non-executable job planning. - Added job orchestration plan tests. - Added CLI tests for valid and invalid job planning. - Files created: `src/services/job-orchestration-plan.ts`, `src/cli/commands/job-plan-command.ts`, `tests/job-orchestration-plan.test.ts`. #### Changed - Registered `jobs:plan` in the default CLI registry, command list, and help output. - Aligned job domain metadata typing with strict Zod optional inference. - Updated README, CLI documentation, job contract documentation, services folder documentation, and task tracking. - Files modified: `src/models/job.ts`, `src/cli/app.ts`, `src/services/README.md`, `tests/cli.test.ts`, `docs/cli.md`, `docs/job-contracts.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: dry-run only; no job execution path, runtime network listener, direct Twitter/X API call, endpoint catalog, browser navigation, login, collector implementation, parser implementation, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: define fixture conventions or approval gates for future execution boundaries. ### 2026-07-04 #### Added - Added `jobs:validate` CLI command for local job definition validation. - Added CLI tests for valid job files, invalid job files, and missing file option handling. - Added job validation CLI documentation. - Files created: `src/cli/commands/job-validate-command.ts`. #### Changed - Registered `jobs:validate` in the default CLI registry, command list, and help output. - Updated README, CLI documentation, job contract documentation, and task tracking. - Files modified: `src/cli/app.ts`, `tests/cli.test.ts`, `docs/cli.md`, `docs/job-contracts.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: validation-only; no job execution path, runtime network listener, direct Twitter/X API call, endpoint catalog, browser navigation, login, collector implementation, parser implementation, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: define fixture conventions or add dry-run orchestration planning without execution. ### 2026-07-04 #### Added - Added `network-api:validate-rules` CLI command for local Network API capture rule validation. - Added parsing support for one rule, an array of rules, or `{ "rules": [...] }` rule files. - Added CLI tests for valid rule files, invalid HTML/document rule files, and missing file option handling. - Files created: `src/cli/commands/network-api-rules-validate-command.ts`. #### Changed - Registered `network-api:validate-rules` in the default CLI registry, command list, and help output. - Expanded Network API capture schema utilities with multi-rule file parsing. - Updated CLI, README, Network API capture documentation, and task tracking. - Files modified: `src/cli/app.ts`, `src/browser/network-api-capture.schema.ts`, `tests/cli.test.ts`, `tests/network-api-capture-schema.test.ts`, `docs/cli.md`, `docs/network-api-capture.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: validation-only; no runtime network listener, direct Twitter/X API call, endpoint catalog, browser navigation, login, collector implementation, parser implementation, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: add a validation-only CLI command for job definition JSON files, or define fixture conventions for Network API capture rules. ### 2026-07-04 #### Added - Added Network API capture TypeScript contracts. - Added Zod validation schemas for capture rules, request snapshots, response snapshots, and capture matches. - Added Network API capture schema tests. - Added Network API capture design documentation. - Files created: `src/browser/network-api-capture.ts`, `src/browser/network-api-capture.schema.ts`, `docs/network-api-capture.md`, `tests/network-api-capture-schema.test.ts`. #### Changed - Updated scraping-adjacent design to specify JSON Network API capture instead of raw page acquisition. - Updated browser folder documentation to mention Network API capture contracts. - Updated `@types/node` from `22.10.2` to `22.12.0` to satisfy Vite peer dependency expectations during install. - Updated `TASKS.md` with Session 16 scope and status. - Files modified: `package.json`, `package-lock.json`, `src/browser/README.md`, `docs/twitter-x-scraping-design.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - Updated dependencies: `@types/node` `22.10.2` to `22.12.0`. - New environment variables: None. - New npm scripts: None. - Known limitations: validation-only; no runtime network listener, direct Twitter/X API call, endpoint catalog, browser navigation, login, collector implementation, parser implementation, HTML scraping, or Twitter/X network behavior has been added. - Next recommended session: add a validation-only CLI command for job definition JSON files, or add validation-only Network API capture rule fixtures. ### 2026-07-04 #### Added - Added job definition TypeScript contracts. - Added Zod validation schemas for job target, limits, output, and metadata. - Added job schema tests. - Added job contract documentation. - Files created: `src/models/job.schema.ts`, `docs/job-contracts.md`, `tests/job-schema.test.ts`. #### Changed - Expanded `src/models/job.ts` with job definition, target, limits, and output contracts. - Updated scraping-adjacent design next-session guidance. - Updated `TASKS.md` with Session 15 scope and status. - Files modified: `src/models/job.ts`, `docs/twitter-x-scraping-design.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: validation-only; no job command, execution path, Twitter/X scraping logic, collector implementation, parser implementation, login, browser navigation, or Twitter/X network behavior has been added. - Next recommended session: add a validation-only CLI command for job definition JSON files. ### 2026-07-04 #### Added - Added the first scraping-adjacent design document. - Documented non-implementation boundaries for Twitter/X behavior. - Documented future job, browser context, collector, parser, and storage boundaries. - Documented safety gates required before implementation. - Files created: `docs/twitter-x-scraping-design.md`. #### Changed - Updated `TASKS.md` with Session 14 scope and status. - Files modified: `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: documentation-only; no Twitter/X scraping logic, collector implementation, parser implementation, login, browser navigation, or Twitter/X network behavior has been added. - Next recommended session: define job model contracts and validation schemas only. ### 2026-07-04 #### Added - Added approval-gated storage preparation executor. - Added `storage:prepare` CLI command for explicitly approved storage preparation. - Added tests for approved directory creation actions. - Added CLI test for missing approval rejection. - Files created: `src/cli/commands/storage-prepare-command.ts`. #### Changed - Updated storage preparation planner module to expose apply behavior for approved directory creation. - Updated CLI command registry, help output, and command list to include `storage:prepare`. - Updated `docs/cli.md` and `README.md` with storage preparation usage. - Updated `TASKS.md` with Session 13 scope and status. - Updated storage preparation and CLI tests. - Files modified: `src/storage/storage-preparation-plan.ts`, `src/cli/app.ts`, `tests/storage-preparation-plan.test.ts`, `tests/cli.test.ts`, `docs/cli.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: `storage:prepare` only applies explicit directory-creation actions after `--apply true`; it does not read scraper data, launch browsers, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: pause infrastructure work and define the first scraping-adjacent design document before adding any Twitter/X behavior. ### 2026-07-04 #### Added - Added dry-run storage preparation planner. - Added `storage:plan` CLI command for reporting filesystem preparation actions without applying changes. - Added tests for no-op, create-directory, and manual-intervention plan outcomes. - Added CLI test for dry-run storage planning. - Files created: `src/storage/storage-preparation-plan.ts`, `src/cli/commands/storage-plan-command.ts`, `tests/storage-preparation-plan.test.ts`. #### Changed - Updated CLI command registry, help output, and command list to include `storage:plan`. - Updated `docs/cli.md` and `README.md` with storage planning usage. - Updated `TASKS.md` with Session 12 scope and status. - Updated CLI tests for storage planning output. - Files modified: `src/cli/app.ts`, `tests/cli.test.ts`, `docs/cli.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: `storage:plan` is dry-run only; it does not create directories, read scraper data, write files, launch browsers, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add explicit approval-gated storage preparation command, or pause infrastructure work and define the first scraping-adjacent design document. ### 2026-07-04 #### Added - Added storage readiness diagnostic service. - Added `storage:diagnose` CLI command for metadata-only storage inspection. - Added tests for ready, missing, and non-directory storage paths. - Added CLI test for storage diagnostics. - Files created: `src/storage/storage-diagnostics.ts`, `src/cli/commands/storage-diagnose-command.ts`, `tests/storage-diagnostics.test.ts`. #### Changed - Updated CLI command registry, help output, and command list to include `storage:diagnose`. - Updated `docs/cli.md` and `README.md` with storage diagnostics usage. - Updated `TASKS.md` with Session 11 scope and status. - Updated CLI tests for storage diagnostics output. - Files modified: `src/cli/app.ts`, `tests/cli.test.ts`, `docs/cli.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: `storage:diagnose` inspects metadata and permissions only; it does not create directories, read scraper data, write files, launch browsers, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add optional dry-run filesystem preparation planning without applying changes. ### 2026-07-04 #### Added - Added sanitized runtime configuration formatter. - Added `config:inspect` CLI command for safe operational configuration inspection. - Added sanitized configuration tests. - Files created: `src/config/sanitized-config.ts`, `src/cli/commands/config-inspect-command.ts`, `tests/config.test.ts`. #### Changed - Updated CLI command registry, help output, and command list to include `config:inspect`. - Updated `docs/cli.md` and `README.md` with config inspection usage. - Updated `TASKS.md` with Session 10 scope and status. - Updated CLI tests for sanitized config output. - Files modified: `src/cli/app.ts`, `src/config/env.ts`, `tests/cli.test.ts`, `docs/cli.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - Fixed boolean environment parsing so `BROWSER_HEADLESS=false` is handled as `false`. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: `config:inspect` prints only safe configuration fields; this session does not launch browsers, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add operational diagnostics for filesystem/storage readiness without reading or writing scraper data. ### 2026-07-04 #### Added - Added command-specific option parsing for `runtime:check`. - Added `--min-node-major ` for runtime check Node.js baseline validation. - Added tests for command-specific option success and validation failure. #### Changed - Updated global CLI option parsing to forward unknown command-specific options to commands. - Updated runtime check logic to use a configurable minimum Node.js major version. - Updated `docs/cli.md` and `README.md` with runtime check option usage. - Updated `TASKS.md` with Session 09 scope and status. - Updated CLI and smoke tests for the configurable Node.js baseline. - Files modified: `src/cli/option-parser.ts`, `src/cli/commands/runtime-check-command.ts`, `src/runtime/runtime-checks.ts`, `tests/cli.test.ts`, `tests/smoke.test.ts`, `docs/cli.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - Runtime Node.js version checks now receive their baseline through typed options instead of relying only on an internal constant. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: only `runtime:check` has command-specific options; this session does not launch browsers, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add operational configuration inspection command that prints sanitized config only. ### 2026-07-04 #### Added - Added typed CLI option parsing and validation. - Added support for `--format json|pretty`. - Added support for `--timestamp true|false`. - Added tests for typed output options and invalid option handling. - Files created: `src/cli/option-parser.ts`. #### Changed - Updated CLI app flow to parse global output options before command execution. - Updated CLI output stringification to support compact and pretty JSON formats. - Updated `docs/cli.md` and `README.md` with CLI option usage. - Updated `TASKS.md` with Session 08 scope and status. - Updated CLI and smoke tests for output formatting behavior. - Files modified: `src/cli/app.ts`, `src/cli/command-registry.ts`, `src/runtime/cli-output.ts`, `tests/cli.test.ts`, `tests/smoke.test.ts`, `docs/cli.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - CLI output formatting now separates JSON envelope creation from output style selection. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: CLI options are global and limited to output formatting; they do not launch browsers, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add command-specific option schemas for future non-scraping operational commands. ### 2026-07-04 #### Added - Added CLI command registry and command name parser. - Added main CLI entrypoint for registered commands. - Added `runtime:check`, `commands`, and `help` command flows. - Added tests for default command execution, help output, and unknown command errors. - Added `npm run cli` script. - Files created: `src/cli/command-registry.ts`, `src/cli/app.ts`, `src/cli/index.ts`, `src/cli/commands/runtime-check-command.ts`, `src/cli/commands/static-command.ts`, `tests/cli.test.ts`. - New npm scripts: `cli`. #### Changed - Updated `check-runtime` CLI to route through the command registry. - Updated `docs/cli.md` and `README.md` with command registry usage. - Updated `TASKS.md` with Session 07 scope and status. - Files modified: `src/cli/check-runtime.ts`, `package.json`, `docs/cli.md`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - Runtime check command execution now uses a reusable CLI command registry. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - Known limitations: CLI registry supports simple command names only and does not include option or flag parsing yet; it does not launch browsers, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add typed CLI options and validation for non-scraping operational commands. ### 2026-07-04 #### Added - Added shared CLI output formatting helpers for success and error responses. - Added shared CLI output stringification helper. - Added component-scoped logger helper for structured logging context. - Added tests for CLI output formatting. - Added `docs/operations.md` to document operational helpers. - Files created: `src/runtime/cli-output.ts`, `src/runtime/logging.ts`, `docs/operations.md`. #### Changed - Updated runtime composition to expose a component-scoped runtime logger. - Updated runtime check CLI to use the shared CLI output envelope and error formatting. - Updated `README.md` and `TASKS.md` with Session 06 scope and status. - Updated smoke tests for CLI output formatting. - Files modified: `src/runtime/app.ts`, `src/cli/check-runtime.ts`, `tests/smoke.test.ts`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - Runtime check CLI output now goes through shared formatting helpers instead of local JSON construction. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - New npm scripts: None. - Known limitations: operations helpers only format output and enrich logging; they do not launch browsers, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add a command registry or CLI argument parser for non-scraping operational commands. ### 2026-07-04 #### Added - Added a runtime check service for Node.js version, health, adapter wiring, and configuration validation. - Added `check-runtime` CLI entrypoint with JSON output and process exit codes. - Added tests for runtime checks without browser navigation or network access. - Added `docs/cli.md` to document the runtime check command. - Files created: `src/runtime/runtime-checks.ts`, `src/cli/check-runtime.ts`, `docs/cli.md`. - New npm scripts: `check:runtime`. #### Changed - Updated `package.json` with the runtime check script. - Updated `README.md` and `TASKS.md` with Session 05 scope and usage. - Updated smoke tests for runtime check coverage. - Files modified: `package.json`, `tests/smoke.test.ts`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - New dependencies: None. - Removed dependencies: None. - New environment variables: None. - Known limitations: runtime checks validate wiring and configuration only; they do not launch a real browser, navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add structured application logging helpers and operational error formatting for future commands, still without scraping behavior. ### 2026-07-04 #### Added - Added `createPlaywrightBrowserPort` as a Playwright-backed implementation of `BrowserPort`. - Added browser lifecycle tests using a fake launcher. - Added browser launch configuration for headless mode and launch timeout. - Added `docs/browser-lifecycle.md` to document the non-scraping browser boundary. - Files created: `src/browser/playwright-browser.ts`, `docs/browser-lifecycle.md`. - New environment variables: `BROWSER_HEADLESS`, `BROWSER_LAUNCH_TIMEOUT_MS`. #### Changed - Updated `.env.example` with browser lifecycle settings. - Updated runtime composition to expose a browser lifecycle adapter. - Updated smoke and infrastructure tests for the browser adapter. - Updated `README.md` and `TASKS.md` with Session 04 scope and status. - Files modified: `.env.example`, `src/config/env.ts`, `src/runtime/app.ts`, `tests/smoke.test.ts`, `tests/infrastructure.test.ts`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: copy `BROWSER_HEADLESS=true` and `BROWSER_LAUNCH_TIMEOUT_MS=30000` into existing local `.env` files if they were created before Session 04. - New dependencies: None. - Removed dependencies: None. - New npm scripts: None. - Known limitations: browser lifecycle support only launches and closes sessions; it does not navigate, authenticate, collect, parse, or scrape Twitter/X. - Next recommended session: add a CLI entrypoint that runs health checks and dependency checks only, without navigation or collection. ### 2026-07-04 #### Added - Added `createAxiosHttpClient` as a generic axios-backed implementation of `HttpClientPort`. - Added `createJsonFileRepository` as a generic filesystem JSON implementation of `RepositoryPort`. - Added shared JSON value types for adapter serialization. - Added runtime configuration values for HTTP timeout and storage directory. - Added infrastructure adapter tests. - Added `docs/infrastructure.md` for adapter behavior and limitations. - Files created: `src/services/axios-http-client.ts`, `src/storage/json-file-repository.ts`, `src/types/json.ts`, `tests/infrastructure.test.ts`, `docs/infrastructure.md`. - New environment variables: `HTTP_TIMEOUT_MS`, `STORAGE_DIR`. #### Changed - Updated `.env.example` with infrastructure adapter settings. - Updated `.gitignore` to exclude local `data/` storage. - Updated runtime composition to expose an HTTP client adapter. - Updated smoke tests to cover new configuration values. - Updated `README.md` and `TASKS.md` with Session 03 scope and status. - Files modified: `.env.example`, `.gitignore`, `src/config/env.ts`, `src/runtime/app.ts`, `tests/smoke.test.ts`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: copy `HTTP_TIMEOUT_MS=30000` and `STORAGE_DIR=./data` into existing local `.env` files if they were created before Session 03. - New dependencies: None. - Removed dependencies: None. - New npm scripts: None. - Known limitations: JSON file storage does not include file locking or concurrent write coordination; no Twitter/X scraping logic, collector implementation, parser implementation, or login flow has been added in Session 03. - Next recommended session: add a controlled browser lifecycle adapter or CLI entrypoint that verifies runtime dependencies without navigating to Twitter/X or collecting data. ### 2026-07-04 #### Added - Added domain model placeholders for source references, job metadata, and health reports. - Added shared `Result` helpers and `AppError` primitives. - Added Clean Architecture ports for browser sessions, repositories, HTTP clients, and collectors. - Added a runtime health-check service and wired it into `createAppRuntime`. - Added `docs/architecture.md` to document Session 02 boundaries. - Files created: `src/types/result.ts`, `src/types/errors.ts`, `src/models/source.ts`, `src/models/job.ts`, `src/models/health.ts`, `src/browser/browser-port.ts`, `src/storage/repository-port.ts`, `src/services/http-client-port.ts`, `src/collectors/collector-port.ts`, `src/services/health-service.ts`, `docs/architecture.md`. - New environment variables: `APP_NAME`. #### Changed - Updated `.env.example` with `APP_NAME`. - Updated runtime startup logging to include the health-check result. - Updated tests to cover the expanded config/runtime contract. - Updated `README.md` and `TASKS.md` with Session 02 scope and status. - Files modified: `.env.example`, `src/config/env.ts`, `src/runtime/app.ts`, `src/index.ts`, `tests/smoke.test.ts`, `README.md`, `TASKS.md`, `CHANGELOG.md`. #### Fixed - None. #### Refactored - Runtime composition now exposes application services through the `AppRuntime` interface. #### Notes - Breaking changes: None. - Migration notes: copy `APP_NAME=twitter-scraper` into existing local `.env` files if they were created before Session 02. - New dependencies: None. - Removed dependencies: None. - New npm scripts: None. - Known limitations: no Twitter/X scraping logic, collector implementation, parser implementation, or login flow has been added in Session 02. - Next recommended session: add concrete non-scraping infrastructure adapters or clarify the first approved adapter scope before any Twitter/X-specific collection work. ### 2026-07-04 #### Added - Created the initial Node.js 22 TypeScript project foundation. - Added Clean Architecture folder structure under `src/`. - Added README documentation for every required source, test, docs, docker, and scripts folder. - Added strict TypeScript, ESLint, Prettier, and Vitest configuration. - Added baseline runtime/config placeholders using dotenv, Zod, and Pino. - Added Dockerfile, docker-compose.yml, MIT license, project README, and TASKS.md. - Files created: `package.json`, `package-lock.json`, `tsconfig.json`, `eslint.config.js`, `prettier.config.js`, `vitest.config.ts`, `.gitignore`, `.prettierignore`, `.env.example`, `Dockerfile`, `docker-compose.yml`, `README.md`, `CHANGELOG.md`, `TASKS.md`, `LICENSE`, `src/**/README.md`, `src/index.ts`, `src/config/env.ts`, `src/runtime/app.ts`, `tests/smoke.test.ts`, `docs/README.md`, `docker/README.md`, `scripts/README.md`. - New dependencies: `axios@1.18.1`, `dotenv@16.4.7`, `pino@9.6.0`, `playwright@1.61.1`, `zod@3.24.1`. - New development dependencies: `@eslint/js@9.39.4`, `@types/node@22.10.2`, `eslint@9.39.4`, `globals@15.14.0`, `prettier@3.4.2`, `typescript@5.7.2`, `typescript-eslint@8.46.0`, `vitest@4.1.9`. - New npm scripts: `build`, `clean`, `format`, `format:check`, `lint`, `test`, `typecheck`. - New environment variables: `NODE_ENV`, `LOG_LEVEL`. #### Changed - None. #### Fixed - None. #### Refactored - None. #### Notes - Breaking changes: None. - Migration notes: None. - Removed dependencies: None. - Known limitations: no Twitter/X scraping logic, collector implementation, parser implementation, or login flow has been added in Session 01. - Next recommended session: define the approved Session 02 scope before adding domain contracts or any scraping-related behavior.