Changes between Version 6 and Version 7 of administration/search

Timestamp:

Apr 17, 2009, 2:51:38 PM (13 years ago)

Author:

hank

Comment:

--

administration/search

@@ -167 +167 @@
 }}}
 
-== Check for results using NutchBean ==
+== Check for results using !NutchBean ==
 {{{
 nutch org.apache.nutch.searcher.NutchBean <Search Term>
@@ -187 +187 @@
   }}}
 
- 1. Create needle.txt in / and then create a symlink to it in /usr/share/doc/wget/
+ 2. Create needle.txt in / and then create a symlink to it in /usr/share/doc/wget/
  {{{
 Symlink to /needle.txt
@@ -199 +199 @@
   }}}
 
- 1. Create jump.txt in /usr/share/
+ 3. Create jump.txt in /usr/share/
   {{{
 nutch... you naughty crawler!
@@ -207 +207 @@
 Do a crawl on the /usr/doc/share:
  1. Edit crawl-urlfilter.txt and regex-urlfilter.txt 
-    Change top level directory to : 
-    +^file:///usr/share/doc/
-    +^file:/usr/share/doc/
- 1. Create file docurls with:
-    file:///usr/share/doc/
+    Change top level directory filter to: 
+    {{{ +^file:///usr/share/doc/
+    +^file:/usr/share/doc/ }}}
+ 1. Create a file called docurls with this line:
+    {{{ file:///usr/share/doc/ }}}
+ 1. Make sure we are using the unaltered nutch:
+    {{{
+    which nutch
+    }}}
+    Output is /usr/bin/nutch which is a symlink to /usr/local/nutch/bin/nutch.
  1. Run the crawl:
     {{{
     nutch crawl urls.doc -dir crawl.doc > crawl.doc.log
     }}}
+
+== Results of Test ==
+First, looking through the crawl.doc.log we don't see any signs that nutch has crawled either the symlink file "needle.txt" or the
+jump file "jump.txt".  If you search for these two names with the log open in vim, neither name is found. The haystack.txt file
+is found in the log:
+{{{
+1203 fetching file:/usr/share/doc/sed/haystack.txt
+}}}
+From visually scanning the file, it doesn't appear that nutch is crawling any higher directories.
+
+Next, try searching the crawldb using !NutchBean.  Here are the results for a few different searches.  Remember to point the 
+searcher.dir in nutch-site.xml to the crawl directory, crawl.doc.  
+
+Look for haystack:
+{{{
+> nutch org.apache.nutch.searcher.NutchBean haystack9
+Total hits: 1
+ 0 20090417072918/file:/usr/share/doc/sed/haystack.txt
+ ... found the haystack.
+haystack9
+}}}
+ 
+Look for needle:
+{{{
+> nutch org.apache.nutch.searcher.NutchBean needle.txt
+Total hits: 0
+> nutch org.apache.nutch.searcher.NutchBean needle008
+Total hits: 0
+}}}
+
+Look for jump:
+{{{
+>  nutch org.apache.nutch.searcher.NutchBean jump.txt
+Total hits: 0
+>  nutch org.apache.nutch.searcher.NutchBean meatball33
+Total hits: 0
+}}}
+
+In conclusion, it doesn't appear that nutch is interested in the symlinks, nor is it jumping into a higher directory.  
+