History: All-in-one crawl-scrape-parse tool

^ Code: https://gitlab.com/tikiwiki/tiki-crawl ^

See also ((Browser Automation))

Goal: Have an all-in-one tool that can do the following
* Crawl links
* ((Scraper)) (fetch data)
* Parse
** Structured data extraction like https://github.com/crwlrsoft/schema-org
** ((Link checker))
* And be part of an [https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html|End to end (E2E) testing] / [https://en.wikipedia.org/wiki/Smoke_testing_(software)|Smoke Test testing] infrastructure
* Ideally, evolve to later [https://goreplay.org/|capture and replay live HTTP traffic into a test environment in order to continuously test with real data]

We want this both for 
* Internal Tiki URLs
** As anonymous
** As a specific user
* External
** The open web
** Corporate info behind the firewall
** Password-protected sites (ideally)

!! Use cases
* Run on various Tiki as https://dev.tiki.org/Pre-dogfood-servers-for-Tiki-26-release-process
** Test new versions of PHP when ((Using GlitchTip as part of the Tiki development process)). Ref: ((doc:PHP8))
** The risky last part of the migration to [https://www.php-fig.org/psr/psr-12/|PSR-12] (We did all the non risky parts already)
* Facilitate the creation of tests for complex Tiki instances. ((doc:PluginList)) and ((doc:Trackers)) mostly.
* Fetch data from external sites for upcoming Tiki AI Chatbot to be able to refer to external sources
* Replace ((doc:PluginCasperJS)) with something modern and supported
* Later, combine with ((doc:Machine Learning))

!! Requirements
* In PHP, so it can be shipped in any Tiki instance


!! Potential tools
!!! PHP
!!! Roach
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/roach-php/widgets/project_factoids_stats?format=js'></script>{HTML}
* https://roach-php.dev/docs/spiders/

!!! Spatie Crawler
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/spatie-crawler/widgets/project_factoids_stats?format=js'></script>{HTML}
* https://github.com/spatie/crawler

!!! Symfony Panther
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/symfony-panther/widgets/project_factoids_stats?format=js'></script>{HTML}
* https://github.com/symfony/panther

!!!! Symfony BrowserKit
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/browser-kit/widgets/project_factoids_stats?format=js'></script>{HTML}
https://github.com/symfony/browser-kit

!!! crwlr
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/crwlr-crawler/widgets/project_factoids_stats?format=js'></script>{HTML}
* https://github.com/crwlrsoft/crawler
* https://www.crwlr.software/blog/good-reasons-to-use-the-crwlr-library

!!! spekulatius/phpscraper
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/phpscraper/widgets/project_factoids_stats?format=js'></script>{HTML}
* https://github.com/spekulatius/phpscraper

!!! Laravel Dusk
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/laravel-dusk/widgets/project_factoids_stats?format=js'></script>{HTML}
* https://packagist.org/packages/laravel/dusk

!!! Codeception
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/codeception/widgets/project_factoids_stats?format=js'></script>{HTML}
https://github.com/Codeception/Codeception

!!! Arachnid
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/arachnid-crawler/widgets/project_factoids_stats?format=js'></script>{HTML}

!!! PHP spider
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/php-spider/widgets/project_factoids_stats?format=js'></script>{HTML}

!!! Acquia BLT (Build and Launch Tool)
This is for Drupal which has a testing component. Adding here for good ideas (ex.: what build blocks do they use?)
{HTML()}<script type='text/javascript' src='https://www.openhub.net/p/acquia_blt/widgets/project_factoids_stats?format=js'></script>{HTML}

!!! Not PHP
* https://github.com/cypress-io/cypress
* https://github.com/microsoft/playwright
* https://github.com/SeleniumHQ/selenium

!! URLs
* https://packagist.org/?query=Scraper
* https://packagist.org/?query=Crawler
* Examples from other communities
** https://gitlab.com/gitlab-org/gitlab-qa
* https://github.com/topics/testing?l=php

~tc~
(alias(All in one exploring crawl-scrape-parse tool)) 
(alias(All-in-one-exploring-crawl-scrape-parse-tool)) 
~/tc~