Vartovii logo

Release notes

Vartovii Changelog

Release archive
3.2.19

Data Quality & Validation System & ABSA Auto-Trigger

Release v3.2.19 focuses on Data Quality & Validation System, ABSA Auto-Trigger, and Reddit Connector.

AddedImprovedFixedInternal
Data Quality & Validation System & ABSA Auto-Trigger

Release digest

Added

  • Data Quality & Validation System 🛡️ - Production-grade data quality infrastructure
    • Entity Type Validation: 68-keyword detection system prevents apartment/product review contamination
      • Apartment keywords: apartment, flat, landlord, rent, tenant (14 total)
      • Product keywords: product, purchase, delivery, shipping, amazon (30 total)
      • Company keywords: employer, workplace, boss, salary, colleagues (24 total)
      • Confidence scoring: ≥3 points → Skip review
      • Integration: Reddit + Google connectors
      • Test accuracy: 100% (6/6 test cases passed)
    • Quality Metrics API: GET /api/admin/data-quality
      • 6 metric categories: summary stats, recent activity, sentiment distribution, duplicates, empty reviews
      • Query params: company_name, days (custom timeframes)
      • Tested with 68K+ reviews across 75 companies
    • Post-Collection Alerting: 4-level severity system (INFO/WARNING/ERROR/CRITICAL)
      • Automatic checks: review count, missing sentiment (>20%/50%), empty reviews (>10%/30%)
      • Structured console logging for easy monitoring
    • Manual Cleanup Tools:
      • DELETE /api/admin/review/{id} - Remove invalid/duplicate reviews
      • POST /api/admin/run-absa?company_name=X - Trigger sentiment analysis
    • Comprehensive Documentation: knowledge-base/docs/data-quality-validation.md
      • System overview with architecture diagram
      • Troubleshooting guide (5 common issues)
      • API reference and best practices

Improved

  • ABSA Auto-Trigger: Replaced subprocess calls with internal HTTP API
    • Old: subprocess.run(["poetry", "run", "python", "-c", "..."]) - silent failures
    • New: httpx.post("/api/admin/run-absa") - proper monitoring
    • Benefits: Better error visibility, no subprocess overhead, easier Cloud Run deployment
  • Reddit Connector: Added entity validation before yielding reviews
    • Tracks validation_skipped_count for monitoring
    • Logs skipped reviews with confidence scores
  • Google Reviews Integration: Added entity validation with place name checking

Fixed

  • Valora Dataset Cleanup: Removed 13 apartment complex reviews (12.6% of Reddit data)
    • Before: 409 reviews (13 invalid apartment reviews)
    • After: 396 clean reviews (0 contamination)
    • Sentiment coverage: 0% → 100%
    • AI Report regenerated with clean data

Internal

  • vartovii/utils/entity_validation.py - Universal validation module
  • vartovii/utils/collection_alerts.py - Post-collection alerting system
  • backend/routers/data_quality.py - Quality metrics API
  • backend/scripts/test_validation.py - Validation test suite
  • backend/scripts/cleanup_valora.py - Data cleanup script
  • knowledge-base/docs/data-quality-validation.md - Public documentation
  • Data Quality: 100% contamination prevention for future scraping jobs
  • Monitoring: Real-time quality metrics for 68K+ reviews
  • ABSA Reliability: API-based triggering with proper error handling
  • Documentation: Self-service troubleshooting guide reduces support burden

Technical notes