【Nutch2.2.1基礎教程之2.1】集成Nutch/Hbase/Solr構建搜索引擎

1、下載相關軟件，并解壓 版本號如下： （1）apache-nutch-2.2.1 （2） hbase-0.90.4 （3）solr-4.9.0 并解壓至/usr/search 2、Nutch的配置 （1）vi /usr/search/apache-nutch-2.2.1/conf/nutch-site.xml propertynamestorage.data.store.class/namevalueorg

1、下載相關軟件，并解壓

版本號如下：

（1）apache-nutch-2.2.1

（2） hbase-0.90.4

（3）solr-4.9.0

并解壓至/usr/search

2、Nutch的配置

（1）vi /usr/search/apache-nutch-2.2.1/conf/nutch-site.xml

storage.data.store.class
org.apache.gora.hbase.store.HBaseStore
Default class for storing data

默認情況下，此語句被注釋掉，將其注釋符號去掉，使其生效。

添加以下語句：

gora.datastore.default=org.apache.gora.hbase.store.HBaseStore

以上三個步驟指定了使用HBase來進行存儲。

以下步驟才是構建基本Nutch的必要步驟。

（4）構建runtime

cd /usr/search/apache-nutch-2.2.1/

ant runtime

（5）驗證Nutch安裝完成

[root@jediael44 apache-nutch-2.2.1]# cd /usr/search/apache-nutch-2.2.1/runtime/local/bin/
[root@jediael44 bin]# ./nutch
Usage: nutch COMMAND
where COMMAND is one of:
inject inject new urls into the database
hostinject creates or updates an existing host table from a text file
generate generate new batches to fetch from crawl db
fetch fetch URLs marked during generate
parse parse URLs marked during fetch
updatedb update web table after parsing
updatehostdb update host table after parsing
readdb read/dump records from page database
readhostdb display entries from the hostDB
elasticindex run the elasticsearch indexer
solrindex run the solr indexer on parsed batches
solrdedup remove duplicates from solr
parsechecker check the parser for a given url
indexchecker check the indexing filters for a given url
plugin load a plugin and run one of its classes main()
nutchserver run a (local) Nutch server on a user defined port
junit runs the given JUnit test
or
CLASSNAME run the class named CLASSNAME
Most commands print help when invoked w/o parameters.

（6）vi /usr/search/apache-nutch-2.2.1/runtime/local/conf/nutch-site.xml 添加搜索任務

http.agent.name
My Nutch Spider

cd /usr/search/apache-nutch-2.2.1/runtime/local/bin/

vi seed.txt

http://nutch.apache.org/

（8）修改網頁過濾器 vi /usr/search/apache-nutch-2.2.1/conf/regex-urlfilter.txt

vi /usr/search/apache-nutch-2.2.1/conf/regex-urlfilter.txt

將

# accept anything else
+.

修改為

# accept anything else
+^http://([a-z0-9]*\.)*nutch.apache.org/

(9)增加索引內容

默認情況下，schema.xml文件中的core及index-basic中的field才會被索引，為索引更多的field，可以通過以下方式添加。

修改nutch-default.xml，新增以下紅色內容

plugin.includes

protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic|index-anchor|index-more|languageidentifier|subcollection|feed|creativecommons|tld

Regular expression naming plugin directory names to

include. Any plugin not matching this expression is excluded.

In any case you need at least include the nutch-extensionpoints plugin. By

default Nutch includes crawling just HTML and plain text via HTTP,

and basic indexing and search plugins. In order to use HTTPS please enable

protocol-httpclient, but be aware of possible intermittent problems with the

underlying commons-httpclient library.

或者可以在nutch-site.xml中添加plugin.includes屬性，并將上述內容復制過去。注意，在nutch-site.xml中的屬性會代替nutch-default.xml中的屬性，因此必須將原有的屬性也復制過去。

3、Hbase的配置

（1）vi /usr/search/hbase-0.90.4/conf/hbase-site.xml

hbase.rootdir



hbase.zookeeper.property.dataDir

默認值為：

 
 hbase.rootdir
 file:///tmp/hbase-${user.name}/hbase
 The directory shared by region servers and into
 which HBase persists. The URL should be 'fully-qualified'
 to include the filesystem scheme. For example, to specify the
 HDFS directory '/hbase' where the HDFS instance's namenode is
 running at namenode.example.org on port 9000, set this value to:
 hdfs://namenode.example.org:9000/hbase. By default HBase writes
 into /tmp. Change this configuration else all data will be lost
 on machine restart.
 
 

但是建議還是把這些屬性做好配置，尤其是第二個關于zoopkeeper的，否則會導致各種問題。以下將目錄配置在本地文件系統(tǒng)中。

hbase.rootdir
file:///home/jediael/hbaserootdir


hbase.zookeeper.property.dataDir
file:///home/jediael/hbasezookeeperdataDir

注意，若無前綴file://，則默認是hdfs://

但在0.90.4版本，默認還是本地文件系統(tǒng)。

4、Solr的配置

（1）覆蓋solr的schema.xml文件。（對于solr4，應該使用schema-solr4.xml)

cp /usr/search/apache-nutch-2.2.1/conf/schema.xml /usr/search/solr-4.9.0/example/solr/collection1/conf/

（2）若使用solr3.6，則至此已經完成配置，但使用4.9，需要修改以下配置：

修改上述復制過來的schema.xml文件

刪除：

增加：

5、啟動抓取任務

（1）啟動HBase

[root@jediael44 bin]# cd /usr/search/hbase-0.90.4/bin/
[root@jediael44 bin]# ./start-hbase.sh

（2）啟動Solr

[root@jediael44 bin]# cd /usr/search/solr-4.9.0/example/
[root@jediael44 example]# java -jar start.jar

（3）啟動Nutch，開始抓取任務

[root@jediael44 example]# cd /usr/search/apache-nutch-2.2.1/runtime/local/bin/
[root@jediael44 bin]# ./crawl seed.txt TestCrawl http://localhost:8983/solr 2

大功告成，任務開始執(zhí)行。

關于上述過程的一些分析請見：

集成Nutch/Hbase/Solr構建搜索引擎之二：內容分析

http://blog.csdn.net/jediael_lu/article/details/37738569

使用crontab來設置Nutch的例行任務時，出現(xiàn)以下錯誤

JAVA_HOME is not set。

于是創(chuàng)建了一個腳本，用于執(zhí)行抓取工作：

#!/bin/bash
export JAVA_HOME=/usr/java/jdk1.7.0_51
/opt/jediael/apache-nutch-2.2.1/runtime/local/bin/crawl /opt/jediael/apache-nutch-2.2.1/runtime/local/urls/ mainhttp://localhost:8080/solr/ 2 >> ~jediael/nutch.log

然后再配置例行任務

30 0,6,8,10,12,14,16,18,20,22 * * * bash /opt/jediael/apache-nutch-2.2.1/runtime/local/bin/myCrawl.sh

聲明：本網頁內容旨在傳播知識，若有侵權等問題請及時與本網聯(lián)系，我們將在第一時間刪除處理。TEL:177 7030 7066 E-MAIL:11247931@qq.com