包简介

Web scraping library

README 文档

README

Sinama is a simple web scraping library.

Requirements

PHP 7.0

Installation

composer require rafaelglikis/sinama

Usage

Create a Sinama Client (which extends Goutte\Client):

use  Sinama\Client;
$client = new Client();

Make requests with the request() method:

// Go to the motherfuckingwebsite.com website
$crawler = $client->request('GET', 'https://motherfuckingwebsite.com/');

The method returns a Crawler object (which extends Symfony/Component/DomCrawler/Crawler).

To use your own Guzzle settings, you may create and pass a new Guzzle 6 instance to Sinama Client. For example, to add a 60 second request timeout:

use  Sinama\Client;
use GuzzleHttp\Client as GuzzleClient;

$client = new Client(new GuzzleClient([
    'timeout' => 60
]));
$crawler = $client->request('GET', 'https://github.com/trending');

For more options visit Guzzle Documentation.

Click on links:

$link = $crawler->selectLink('PHP')->link();
$crawler = $client->click($link);
echo $crawler->getUri()."\n";

Extract data the symfony way:

$crawler->filter('h3 > a')->each(function ($node) {
    print trim($node->text())."\n";
});

Or use Sinama special methods:

$crawler = $client->request('GET', 'https://github.com/trending');
echo '<html>';
echo '<head>';
echo '<title>'.$crawler->findTitle().'</title>';
echo '<head>';
echo '<body>';
echo '<h1>'.$crawler->findTitle().'</h1>';
echo '<p>Main Image: '.$crawler->findMainImage().'</p>';
echo $crawler->findMainContent();
echo '<pre>';
echo 'Links: ';
print_r($crawler->findLinks());
echo 'Emails: ';
print_r($crawler->findEmails());
echo 'Images: ';
print_r($crawler->findImages());
echo '</pre>';
echo '</body>';
echo '</html>';

Submit forms:

$crawler = $client->request('GET', 'https://www.google.com/');
$form = $crawler->selectButton('Google Search')->form();
$crawler = $client->submit($form, ['q' => 'rafaelglikis/sinama']);
$crawler->filter('h3 > a')->each(function ($node) {
    print trim($node->text())."\n";
});

Now that we have learned enough let's scrape a site with Sinama Spider:

use Sinama\Crawler;
use Sinama\Spider as BaseSpider;

class Spider extends BaseSpider
{
    public function parse(Crawler $crawler)
    {
        $crawler->filter('div.read-more > a')->each(function (Crawler $node) {
            $this->scrape($node->attr('href'));
        });

        $crawler->filter('div.blog-pagination > a')->each(function ($node) {
            $this->follow($node->attr('href'));
        });
    }

    public function scrape($url)
    {
        echo "*************************************************** ".$url."\n";
        $crawler = $this->client->request('GET', $url);
        echo "Title: " . $crawler->findTitle() . "\n";
        echo "Main Image: " . $crawler->findMainImage()."\n";
        echo "Main Content: \n" . $crawler->findMainContent()."\n";
        echo "Emails: \n";
        print_r($crawler->findEmails());
        echo "Links: \n";
        print_r($crawler->findLinks());
    }

    public function getStartUrls(): array
    {
        return [
            'https://blog.scrapinghub.com'
        ];
    }
}

$spider = new Spider([
    'start_urls' => [ 'https://blog.scrapinghub.com' ],
    'max_depth' => 2,
    'verbose' => true
]);
$spider->run();

TODO

Crawler::findTags()

rafaelglikis/sinama 适用场景与选型建议

rafaelglikis/sinama 是一款基于 PHP 开发的 Composer 扩展包，目前已累计 16 次下载、GitHub Stars 达 3，最近一次更新时间为 2018 年 08 月 15 日，在 PHP 生态内属于活跃度较高的组件。

它主要适用于以下技术方向：「scraper」「crawler」「crawling」「scraping」等业务场景。在实际项目中，围绕这些方向常见需要落地的问题包括：接口对接、性能调优、并发安全、与既有框架（Laravel / ThinkPHP / Yii / Webman 等）的兼容适配，以及生产环境的日志埋点与稳定性保障。

我们在过去多个企业项目中使用过 rafaelglikis/sinama 或与其功能相近的方案，如果你在选型或落地过程中遇到问题，例如 版本兼容、二次改造、私有化封装、与内部系统对接、生产 BUG 排查，欢迎联系我们协助评估。

围绕 rafaelglikis/sinama 我们能提供哪些服务？

定制开发 / 二次开发

基于 rafaelglikis/sinama 在你已有业务上做功能扩展、字段裁剪、UI 适配、与内部账号 / 权限 / 日志系统的深度对接。

BUG 修复 & 性能优化

线上偶发问题、内存泄漏、慢查询、并发异常等排查修复；针对高流量场景做缓存、队列、索引层面的调优。

项目外包 & 长期维护

承接完整的项目从需求 → 设计 → 开发 → 上线 → 长期运维；也可按月提供技术保姆服务。

yvsm@zunyunkeji.com QQ：316430983 微信：yvsm316 西安尊云信息科技 · 专注 PHP / Go / 分布式系统研发

与 rafaelglikis/sinama 相关的其它包

同方向 / 同关键字的高下载量 PHP Composer 包推荐，方便对比选型：

eddieace/php-simple 13

crawlbase/crawlbase 16

A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API

plasticstudio/silverstripe-seo 2

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content (forked from Cyber-Duck/Silverstripe-SEO)

crawlzone/crawlzone 82

Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web search and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing.

web-vision/ai-llms-txt 3

TYPO3 extension for generating .well-known/llm.txt links according to llmstxt.org specification

hypejunction/http-parser 5

A tool for scraping URL resources (oEmbed, OpenGraph, Twitter cards, JSON-LD)

rafaelglikis/sinama

包简介

关键字：