# Changelog of Dailymotion Download & Text & Metadata Scraper (`ilborso/dailymotion-download-text-metadata-scraper`) Actor

- **URL**: https://apify.com/ilborso/dailymotion-download-text-metadata-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/ilborso/dailymotion-download-text-metadata-scraper.md

## Changelog

All notable changes to the Dailymotion Transcript Scraper project are documented here.

### \[0.4.0] - 2026-07-29 - Parallel Video Processing

#### Added

- **Parallel video processing** via `PARALLEL` environment variable
  - Control the number of concurrent video operations
  - Configurable per run via environment variable (Apify)
  - CLI argument `--parallel` for standalone mode
  - Default: 1 (sequential processing)

- **Thread pool executor** for efficient parallel processing
  - Both direct URL processing and search results benefit from parallelization
  - Thread-safe video data collection
  - Proper error handling in parallel context

- **Performance improvements**
  - Process multiple videos simultaneously
  - Reduce total execution time for batch operations
  - Better resource utilization

#### Implementation Details

- Uses Python's `concurrent.futures.ThreadPoolExecutor`
- Semaphore-based limiting in Apify Actor (async)
- ThreadPoolExecutor-based limiting in standalone CLI (sync)
- Logging includes parallel limit in configuration output

#### Usage Examples

**Apify Actor** - Set environment variable:

```
PARALLEL=4
```

**CLI** - Use command line argument:

```bash
python dailymotion_actor.py --urls "url1" "url2" "url3" --parallel 3 --output results.json
```

**Search with parallel processing:**

```bash
python dailymotion_actor.py --query "python tutorial" --limit 10 --parallel 5 --output results.json
```

#### Backward Compatibility

- ✓ Default sequential behavior (PARALLEL=1 or --parallel 1)
- ✓ No changes to input/output format
- ✓ Thread-safe operations

#### Performance Guidelines

- `PARALLEL=1`: Sequential (default, lowest resource usage)
- `PARALLEL=2-3`: Balanced (recommended for typical runs)
- `PARALLEL=5+`: High concurrency (for batch operations, requires more resources)

***

### \[0.3.0] - 2026-07-29 - Video Download Feature

#### Added

- **Video download functionality** with configurable quality levels
  - `downloadVideo` parameter (true/false, default: false)
  - `quality` parameter with three options:
    - `720p` - Best quality up to 720p resolution
    - `1024` - Best quality up to 1024p resolution
    - `maximum` - Maximum available quality

- **New function `download_video()`** in both `src/main.py` and `dailymotion_actor.py`
  - Downloads video file to temporary directory
  - Returns file path on success
  - Logs file size and completion status
  - Gracefully handles download failures

- **Enhanced output structure**
  - `videoFilePath` - Path to downloaded video file (when enabled)
  - `videoQuality` - Quality level of downloaded video (when enabled)

- **CLI support** (dailymotion\_actor.py)
  - `--download-video true/false` - Enable/disable video download
  - `--quality 720p/1024/maximum` - Set video quality

#### Changed

- **process\_video()** function signature updated with two new parameters:
  - `download_video_flag: bool = False`
  - `video_quality: str = "720p"`
- Enhanced logging with video download status in `[VIDEO]` component
- Configuration logging updated to show download settings

#### Implementation Details

- Video files stored in temporary directory: `<temp>/dm_videos/`
- Format mapping: 720p → `best[height<=720]`, 1024 → `best[height<=1024]`, maximum → `best`
- Uses yt-dlp's format selection for quality control
- Download failures don't stop transcription extraction (graceful degradation)
- File size reported in MB in logs

#### Example Usage

**Apify Actor** (src/main.py):

```json
{
  "urls": ["https://www.dailymotion.com/video/xaeie8a"],
  "downloadVideo": true,
  "quality": "720p"
}
```

**CLI** (dailymotion\_actor.py):

```bash
python dailymotion_actor.py --urls "https://www.dailymotion.com/video/xaeie8a" --download-video true --quality 720p --output results.json
```

#### Backward Compatibility

- ✓ Default behavior unchanged (video download disabled)
- ✓ Existing input/output format still supported
- ✓ All new parameters optional with sensible defaults
- ✓ Download failures don't affect other processing steps

***

### \[0.2.1] - 2025-07-23 - Metadata Extraction Bugfix (Hot Fix)

#### Fixed

- **Metadata extraction failures on Apify platform**
  - yt-dlp return code 1 errors in LIMITED\_PERMISSIONS sandbox
  - Firefox impersonation not available in container
  - stderr output truncation hiding actual error messages
  - No retry logic for transient failures

#### Added

- **Retry logic with exponential backoff** (3 attempts, 2-8s delays)
- **Enhanced yt-dlp flags**:
  - `--socket-timeout 30` - Explicit socket timeout
  - `--user-agent "Mozilla/5.0..."` - Static user-agent (avoids impersonation)
- **Full stderr logging** - No truncation, complete error messages
- **Per-attempt logging** - Shows which attempt succeeded/failed
- **Improved timeout handling** - 90s total (up from 60s)

#### Changed

- `extract_metadata()`: Added retry logic (now ~75 lines vs 20)
- `download_subtitles()`: Added Dailymotion extractor args
- Logging: Full stderr instead of truncated (300 chars)

#### Behavior Changes

- **Transient failures** now retry automatically
- **Permanent failures** fail faster (after 3 attempts instead of hanging)
- **Better diagnostics** - stderr output now shows actual error

#### Documentation

- **BUGFIX\_METADATA\_EXTRACTION.md** - Detailed explanation and testing

#### Backward Compatible

- ✓ Same input/output format
- ✓ Same function signatures
- ✓ All existing tests pass
- ✓ No configuration changes needed

***

### \[0.2.0] - 2025-07-23 - Logging Enhancement

#### Major Changes

- **Added comprehensive logging system** across all components
- **Enhanced observability** for production monitoring and troubleshooting
- **Extensive documentation** for all audiences (users, developers, operators)

#### Added - Code

- **src/main.py**: Added 80+ log statements with 8 component prefixes
  - `[SEARCH]` - Video discovery logging
  - `[METADATA]` - Metadata extraction logging
  - `[SUBTITLES]` - Subtitle download logging
  - `[PARSE_SRT]` - SRT parsing logging
  - `[PARSE_VTT]` - VTT parsing logging
  - `[LANG_SELECT]` - Language selection logging
  - `[VIDEO]` - Per-video orchestration logging
  - `[MAIN]` - Job orchestration logging

- **src/logger\_config.py**: New extensible logging configuration module
  - Environment-based log level control (`ACTOR_LOG_LEVEL`)
  - Visual markers and formatting constants
  - Component prefix definitions
  - Extensible for future enhancements (metrics, JSON logging, etc.)

#### Added - Documentation

- **LOGGING.md** (13 KB) - Main logging guide with examples and patterns
- **LOGGING\_COMPLETE.md** (14 KB) - Complete implementation details
- **PROJECT\_STRUCTURE.md** (13 KB) - Project organization and file purposes
- **INDEX\_DOCUMENTATION.md** (new) - Documentation roadmap
- **ENHANCEMENT\_SUMMARY.md** (new) - Summary of changes and metrics
- **.actor/LOGGING\_GUIDE.md** (9 KB) - Technical logging patterns reference
- **.actor/LOGGING\_ENHANCEMENTS.md** (9.4 KB) - Feature details
- **.actor/TROUBLESHOOTING\_LOGS.md** (9.9 KB) - Problem/solution pairs
- **.actor/LOGGING\_SUMMARY.md** (11 KB) - Changes summary
- **CHANGELOG.md** (this file) - Change tracking

#### Enhanced - Code Structure

- **Main function**: Now includes detailed job orchestration logs
- **Search function**: Logs query, execution, JSON parsing, results
- **Metadata extraction**: Logs extraction process and key fields
- **Subtitle download**: Logs file operations and parsing
- **Language selection**: Logs available/preferred languages and selection logic
- **Error handling**: All exceptions now logged with context

#### Enhanced - User Experience

- Clear execution flow with phase separators
- Visual markers (✅ ❌ ⚠️ 🛑 ✓) for quick scanning
- Component-level filtering for focused log analysis
- Four log levels for flexibility (DEBUG/INFO/WARNING/ERROR)
- Extensible logging configuration for future enhancements

#### Changed - Nothing

- ✓ No breaking changes to API or configuration
- ✓ No changes to input/output format
- ✓ No changes to core functionality
- ✓ No performance degradation (<1% overhead)

#### Behavior - Enhanced

- **Visibility**: Every phase of execution now has detailed logging
- **Traceability**: Can track individual videos through processing pipeline
- **Debuggability**: Easy to identify where failures occur
- **Monitorability**: Statistics visible in logs for production monitoring
- **Extensibility**: Configuration ready for metrics collection, APM integration

***

### \[0.1.0] - 2025-07-23 - Initial Apify Actor Release

#### Added - Initial Release

- Complete Apify actor for Dailymotion transcript scraping
- Support for direct URLs and search queries
- Multi-language subtitle/caption support
- Rich metadata extraction via yt-dlp
- Dataset and KV store output

#### Features

- **Input support**:
  - Direct Dailymotion video URLs
  - Search queries with time filters
  - Language preference configuration
  - Customizable output file naming

- **Processing capabilities**:
  - Metadata extraction (30+ fields)
  - Manual subtitle download
  - Auto-caption download
  - Multi-language transcript selection
  - SRT/VTT parsing

- **Output format**:
  - Per-video dataset items
  - Aggregated JSON in KV store
  - Summary statistics
  - Language coverage reporting

#### Architecture

- Async/await for Apify compatibility
- Cross-platform path handling
- UTF-8 encoding support
- Error recovery and continuation
- Temporary file cleanup

#### Documentation (Initial)

- README.md - Main documentation
- Dockerfile - Container definition
- requirements.txt - Dependencies
- Input/output/dataset schemas

***

### Logs & Behavior Overview

#### Execution Phases

##### 1. Job Initialization

- Reads input configuration
- Validates parameters
- Logs configuration summary
- Initializes temporary directories

**Typical logs:**

```
[MAIN] Configuration:
[MAIN]   - Direct URLs: 1
[MAIN]   - Search query: (none)
[MAIN]   - Preferred languages: ['en', 'it']
```

##### 2. URL Discovery

- Processes direct URLs
- Executes search if query provided
- Deduplicates results
- Reports total URLs to process

**Typical logs:**

```
[SEARCH] Starting search for query: 'python'
[SEARCH] Found 10 video URL(s)
[MAIN] Total unique URLs to process: 11
```

##### 3. Per-Video Processing Loop

For each video:

**3a. Metadata Extraction**

- Runs yt-dlp with JSON output
- Extracts title, duration, views, availability
- Logs available subtitles and captions

**Typical logs:**

```
[METADATA] Starting metadata extraction
[METADATA] Extracted: title='Python Tutorial', views=50000
[METADATA] Subtitles available: ['en', 'fr', 'it']
```

**3b. Subtitle Download**

- Downloads manual subtitles
- Downloads auto-captions
- Tracks file count and types

**Typical logs:**

```
[SUBTITLES] Found 2 SRT files and 1 VTT file
[SUBTITLES] ✓ en: 245 lines (SRT)
[SUBTITLES] ✓ it: 238 lines (VTT)
```

**3c. Parsing & Language Selection**

- Parses SRT/VTT files
- Removes timestamps and metadata
- Selects primary language based on preference
- Calculates word count

**Typical logs:**

```
[PARSE_SRT] extracted 187 lines (skipped 63)
[LANG_SELECT] ✓ Selected: en (priority 1), 2847 words
```

**3d. Result Recording**

- Pushes item to dataset
- Tracks success/failure
- Logs title and available languages

**Typical logs:**

```
[MAIN] ✅ [1/10] SUCCESS: Python Tutorial (langs: ['en', 'it'])
```

##### 4. Summary Generation

- Calculates statistics
- Counts successes/failures
- Reports language coverage
- Saves aggregated output to KV store

**Typical logs:**

```
[MAIN] Summary statistics:
[MAIN]   - Succeeded: 9
[MAIN]   - Failed: 1
[MAIN]   - Success rate: 90%
[MAIN]   - Languages found: {'en': 9, 'it': 8, 'fr': 5}
```

##### 5. Job Completion

- Confirms KV store save
- Logs final status
- Reports time and statistics

**Typical logs:**

```
[MAIN] ✅ Summary saved successfully
[MAIN] ✅ JOB COMPLETE
```

#### Error Handling & Recovery

**Single Video Failure:**

- Logged with specific reason
- Job continues to next video
- Counted in failure statistics
- Included in final summary

**Example:**

```
[METADATA] yt-dlp failed with return code 1
[METADATA] stderr: Video unavailable
[MAIN] ❌ [2/10] FAILED: Video processing returned None
```

**No URLs Found:**

- Logged as critical
- Job aborts with warning
- No processing attempted

**Example:**

```
[SEARCH] Found 0 video URL(s)
[MAIN] 🛑 No URLs to process. Aborting.
```

**Language Fallback:**

- Preferred languages not found
- Uses first available language
- Logged as warning, not error
- All languages still included in output

**Example:**

```
[LANG_SELECT] Available languages: ['fr', 'de']
[LANG_SELECT] Preferred languages: ['en', 'it']
[LANG_SELECT] ⚠ No preferred language matched. Using first available: fr
```

#### Performance Characteristics

**Search Phase:**

- Fast: <5 seconds (typical)
- Normal: 5-10 seconds
- Slow: >10 seconds (check network)

**Metadata Extraction:**

- Fast: <5 seconds
- Normal: 5-10 seconds
- Slow: >10 seconds (may timeout at 60s)

**Subtitle Download:**

- No subtitles: <1 second
- Small subtitles: 5-10 seconds
- Large subtitles: 10-30 seconds (may timeout at 120s)

**Per-Video Total:**

- Typical: 10-30 seconds
- With many subtitles: 30-60 seconds
- Slow network: >60 seconds

#### Success Metrics

**Typical successful run:**

```
[MAIN] Summary statistics:
[MAIN]   - Total processed: 5
[MAIN]   - Succeeded: 5
[MAIN]   - Failed: 0
[MAIN]   - Success rate: 100%
[MAIN]   - Languages found: {'en': 5, 'it': 4, 'fr': 3}
```

**Acceptable run (some failures):**

```
[MAIN] - Succeeded: 8
[MAIN] - Failed: 2
[MAIN] - Success rate: 80%
```

**Issues to investigate:**

```
[MAIN] - Succeeded: 2
[MAIN] - Failed: 8
[MAIN] - Success rate: 20%
→ Check: Network, URLs validity, yt-dlp version
```

#### Monitoring Recommendations

**In Production:**

- Monitor `SUCCESS` count ≥ 90%
- Watch for timeout patterns
- Track language coverage
- Alert on `ERROR` in logs
- Monitor job completion status

**Key Statistics to Track:**

- Success rate percentage
- Language coverage
- Videos with no subtitles (expected)
- Failed video reasons
- Processing time trends

**Alert Thresholds:**

- 0 successful videos: CRITICAL
- Success rate <50%: WARNING
- Success rate <20%: CRITICAL
- All videos timeout: CRITICAL

***

### Version History

| Version | Date | Type | Focus |
|---------|------|------|-------|
| 0.2.0 | 2025-07-23 | Enhancement | Comprehensive Logging |
| 0.1.0 | 2025-07-23 | Initial Release | Core Scraper |

***

### Dependencies Unchanged

- `apify>=1.0.0` - Apify SDK
- `yt-dlp>=2024.1.0` - Video extraction

No dependency changes in 0.2.0 release.

***

### Breaking Changes

**None.** Version 0.2.0 is fully backward compatible with 0.1.0.

- ✓ Same input format
- ✓ Same output format
- ✓ Same API
- ✓ Same functionality
- ✓ Same configuration

***

### Next Steps / Roadmap

Potential future enhancements (0.3.0+):

- \[ ] Metrics collection and export
- \[ ] APM (Application Performance Monitoring) integration
- \[ ] JSON structured logging for centralized monitoring
- \[ ] Performance profiling per component
- \[ ] Cost tracking per video
- \[ ] Failure rate alerting
- \[ ] Language coverage analytics
- \[ ] Subtitle quality metrics

***

### Support & Reporting

#### Documentation

- See `LOGGING.md` for logging guide
- See `TROUBLESHOOTING_LOGS.md` for problem solving
- See `PROJECT_STRUCTURE.md` for architecture

#### Issues

1. Check `TROUBLESHOOTING_LOGS.md` for your error pattern
2. Consult `LOGGING.md` for detailed analysis
3. Enable debug mode: `ACTOR_LOG_LEVEL=debug`
4. Capture full logs for analysis

***

Generated: 2025-07-23
Last Updated: 2025-07-23