包简介

An full search page crawler to enable complex and customized searching abilities.

README 文档

README

Crawler

An easy to use full-website page crawler to make provide search results on your page. The crawler module gather all information about the sites on the configured domain and stores the index in the database. From there you can now create search queries to provide search results. There are also helper methods which provide intelligent search results by splitting the input into multiple search queries (used by default).

Installation

Install the module via composer:

composer require luyadev/luya-module-crawler:^3.0

After installation via Composer include the module to your configuration file within the modules section.

'modules' => [
    //...
    'crawler' => [
        'class' => 'luya\crawler\frontend\Module',
        'baseUrl' => 'https://luya.io',
        /*
        'filterRegex' => [
            '#.html#i', // filter all links with `.html`
            '#/agenda#i', // filter all links which contain the word with leading slash agenda,
            '#date\=#i, // filter all links with the word date inside. for example when using an agenda which will generate infinite links
        ],
        'on beforeProcess' => function() {
            // optional add or filter data from the BuilderIndex, which will be processed to the Index afterwards
        },
        'on afterIndex' => function() {
            // optional add or filter data from the freshly built Index
        }
        */
    ],
    'crawleradmin' => 'luya\crawler\admin\Module',
]

Where baseUrl is the domain you want to crawler all information.

After setup the module in your config you have to run the migrations and import command (to setup permissions):

./vendor/bin/luya migrate
./vendor/bin/luya import

Running the Crawler

To execute the command (and run the crawler proccess) use the crawler command crawl, you should put this command in cronjob to make sure your index is up-to-date:

Make sure your page is in utf8 mode (<meta charset="utf-8"/>) and make sure to set the language <html lang="<?= Yii::$app->composition->langShortCode; ?>">.

./vendor/bin/luya crawler/crawl

In order to provide current crawl results you should create a cronjob which crawls the page each night: cd httpdocs/current && ./vendor/bin/luya crawler/crawl

Crawler Arguments

All crawler arguments for crawler/crawl, an example would be crawler/crawl --pdfs=0 --concurrent=5 --linkcheck=0:

name	description	default
linkcheck	Whether all links should be checked after the crawler has indexed your site	true
pdfs	Whether PDFs should be indexed by the crawler or not	true
concurrent	The amount of conccurent page crawles	15

Stats

You can also get statistic results enabling a cronjob executing each week:

./vendor/bin/luya crawler/statistic

Create search form

Make a post request with query to the crawler/default/index route and render the view as follows:

<?php
use luya\helpers\Url;
use yii\widgets\LinkPager;
use luya\crawler\widgets\DidYouMeanWidget;
/* @var $query string The lookup query encoded */
/* @var $language string */
/* @var $this \luya\web\View */
/* @var $provider \yii\data\ActiveDataProvider */
/* @var $searchModel \luya\crawler\models\Searchdata */
?>

<form class="searchpage__searched-form" action="<?= Url::toRoute(['/crawler/default/index']); ?>" method="get">
    <input id="search" name="query" type="search" value="<?= $query ?>">
    <input type="submit" value="Search"/>
</form>

<h2><?= $provider->totalCount; ?> Results</h2>

<?php if ($query && $provider->totalCount == 0): ?>
    <div>No results found for &laquo;<?= $query; ?>&raquo;.</div>
<?php endif; ?>

<?= DidYouMeanWidget::widget(['searchModel' => $searchModel]); ?>
<?php foreach($provider->models as $item): /* @var $item \luya\crawler\models\Index */ ?>
    <h3><?= $item->title; ?></h3>
    <p><?= $item->preview($query); ?></p>
    <a href="<?= $item->url; ?>"><?= $item->url; ?></a>
<?php endforeach; ?>
<?= LinkPager::widget(['pagination' => $provider->pagination]); ?>

Crawler Settings

You can use crawler tags to trigger certains events or store informations:

tag	example	description
CRAWL_IGNORE	`<!-- [CRAWL_IGNORE] -->Ignore this<!-- [/CRAWL_IGNORE] -->`	Ignores a certain content from indexing.
CRAWL_FULL_IGNORE	`<!-- [CRAWL_FULL_IGNORE] -->`	Ignore a full page for the crawler, keep in mind that links will be added to index inside the ignore page.
CRAWL_GROUP	`<!-- [CRAWL_GROUP]api[/CRAWL_GROUP] -->`	Sometimes you want to group your results by a section of a page, in order to let crawler know about the group/section of your current page. Now you can group your results by the `group` field.
CRAWL_TITLE	`<!-- [CRAWL_TITLE]My Title[/CRAWL_TITLE] -->`	If you want to make sure to always use your customized title you can use the CRAWL_TITLE tag to ensure your title for the page:

luyadev/luya-module-crawler 适用场景与选型建议

luyadev/luya-module-crawler 是一款基于 PHP 开发的 Composer 扩展包，目前已累计 33.87k 次下载、GitHub Stars 达 7，最近一次更新时间为 2016 年 03 月 08 日，在 PHP 生态内属于活跃度较高的组件。

它主要适用于以下技术方向：「php」「module」「crawler」「yii」「yii2」「luya」等业务场景。在实际项目中，围绕这些方向常见需要落地的问题包括：接口对接、性能调优、并发安全、与既有框架（Laravel / ThinkPHP / Yii / Webman 等）的兼容适配，以及生产环境的日志埋点与稳定性保障。

我们在过去多个企业项目中使用过 luyadev/luya-module-crawler 或与其功能相近的方案，如果你在选型或落地过程中遇到问题，例如 版本兼容、二次改造、私有化封装、与内部系统对接、生产 BUG 排查，欢迎联系我们协助评估。

围绕 luyadev/luya-module-crawler 我们能提供哪些服务？

定制开发 / 二次开发

基于 luyadev/luya-module-crawler 在你已有业务上做功能扩展、字段裁剪、UI 适配、与内部账号 / 权限 / 日志系统的深度对接。

BUG 修复 & 性能优化

线上偶发问题、内存泄漏、慢查询、并发异常等排查修复；针对高流量场景做缓存、队列、索引层面的调优。

项目外包 & 长期维护

承接完整的项目从需求 → 设计 → 开发 → 上线 → 长期运维；也可按月提供技术保姆服务。

yvsm@zunyunkeji.com QQ：316430983 微信：yvsm316 西安尊云信息科技 · 专注 PHP / Go / 分布式系统研发