scRUBYt! is a simple to learn and use, yet powerful web scraping toolkit written in Ruby. The idea behind making scRUBYt! was to show a few simple concepts of Web extraction as a practical extension of this tutorial.
You will see an example of:
Goal:
Solution:
The code:
require 'rubygems'
require 'scrubyt'
pricegrabber_data = Scrubyt:: Extractor.define do
fetch 'http://cameras.pricegrabber.com'
fill_textfield 'form_keyword', 'canon EOS 20D'
submit
camera 'Canon EOS 20D Digital SLR Camera Body Only', :generalize => false do
camera_detail do
detail_tab 'Product Details', :generalize => false do
detail_detail do
camera_record :generalize => false do
image "http://ai.pricegrabber.com/pi/0/37/48/3748716_125.jpg" do
image_url "src", :type => :attribute do
download 'camera_images', :type => :download
end
end
name 'EOS 20D Digital SLR Camera Body Only'
lowestprice '$649.95'
rating_and_reviews :contains => '(Read 31 Reviews)' do
rating /(d.dd / d.dd)/
end
end
end
end
end
end
end
puts pricegrabber_data.to_xml
If you’d like to run, it cut’n'paste it from here (otherwise the code will be messed up).
If you are more of a visual type, check out this cart depicting the process.
Troubleshooting
Problem: “This extractor does not work!”
Solution: ‘Lowest Price’ changes quite frequently - check if the actual ‘Lowest Price’ at http://cameras.pricegrabber.com/digital/canon/m/3748716/details/ does match the one stated in the extractor. If not, simply “upgrade” the example with the actual correct data. Hmmm, this should work.
Problem: “The extractor returns data for the first listed item only. Where is the rest of data?”
Solution: The extractor was designed only to give a taste of how to solve similar scraping problems like it was stated in ‘Goals” therefore it is limited to the first item only. If you would like to scrape all the items on the list page simply remove :generalize => false from
“camera ‘Canon EOS 20D Digital SLR Camera Body Only’, :generalize => false do”. However, it will return data from the first list page only, while no crawling to the next pages is defined in the scraper (in our case it would be kind of insane since no next pages exist for this type of camera).
July 24th, 2007 at 12:26 pm
This tool sounds like it could be pretty useful and surely nifty.
Nevertheless, you might want to consider using the PriceGrabber lists and price alerts instead. Some engineers have been doing the thinking for you and you won’t need any coding
… But we’re still glad to see you are using the best comparison shopping website as an example!!
July 25th, 2007 at 12:04 am
Thomas,
you have made a good point! It seems insane to mess around with coding when you can simply use price alerts instead. However, the purpose of the pricegrabber scenario was not to gain lowest price of the product in the first place but to give an example how can be similar problems solved with scRUBYt!. Furthermore scRUBYt! is able to scrap much more than basic data (theoretically all the data provided about a given product(s) including images, rating etc.). How I have mentioned it was just a sample scraper, a ‘how to’ for users and was not meant to be reused in its present form.
I will check out the features you have recommended for sure. Thanks for drawing them to my attention!
July 26th, 2007 at 12:15 pm
Hey CopperMonkey -
I’m sure you’d agree, such content should not be used commercially without the permission of the PriceGrabber folks, as I’m sure is described in their legal verbiage on the PriceGrabber site. PriceGrabber certainly appreciates this sort of attention and is quite happy to be used as an example in this context, but needs to maintain its rights too, of course.
Not that I work for them or anything…
July 27th, 2007 at 3:22 am
Hey Darren,
I completely agree :). We do not support any illegal activity including scraping data from sites such as pricegrabber and reuse it for commercial purposes.
I understand that the guys at pricegrabber might have some concerns
I emphasize again that this tutorial serves didactic purposes only and was not meant to be used for ‘evil plots’.
August 17th, 2007 at 5:09 am
Two questions:
1) Is is possible to EXTRACT data from form fields (such as values from hidden fields, etc)? I can do this when I provide an XPath in the production extractor, but I can’t get the learning extractor to accept the XPath.
2) Is it possible for @nextpage@ to work with strictly numbered page results? I’m scraping a page that unfortunately does not have a “Next” link of any kind. Perhaps if @nextpage@ supported a sort of “url” pattern, such as:
@nextpage “http://example.com/index.php?page=NEXTPAGE”@ or something like that.
Thanks for this awesome library, by the way
August 29th, 2007 at 5:40 am
I tried this example and I get the following error:
TypeError: can’t convert Regexp into String
I used the file from “Pastie”
The google example worked just fine!
Thanks
August 31st, 2007 at 10:45 pm
I get that same error. It’s because scrubyt is trying to run a regular expression over a regular expression. This patch fixes the error but I think pricegrabber has updated its site so this script dosn’t work anyway. But now at least it just produces incorrect data rather than bombing out entirely!
Basically, add “filters[0].example.class != Regexp” to pattern.rb line 100:
— /var/lib/gems/1.8/gems/scrubyt-0.3.0/lib/scrubyt/core/scraping/pattern.rb 2007-08-31 13:34:09.000000000 -0700
+++ workspace-RDT/scraper/scrubyt/core/scraping/pattern.rb 2007-08-31 23:38:19.000000000 -0700
@@ -97,7 +97,7 @@
#@generalize was not set up explicitly
if @options[:generalize].nil?
@options[:generalize] = true if parent.nil?
- @options[:generalize] = false if filters[0].example =~ /.+[[a-zA-Z].+]$/
+ @options[:generalize] = false if filters[0].example.class != Regexp && filters[0].example =~ /.+[[a-zA-Z].+]$/
end
September 14th, 2007 at 7:30 am
Rather than changing pattern.rb you could use the ’source’ method of the Regexp class as shown below. It worked for me.
ratingandreviews :contains => ‘(Read 31 Reviews)’ do
rating /(d.dd / d.dd)/.source
end
May 17th, 2008 at 9:26 am
aqund ueblgw
http://virginiaperry.seitenclique.net/modernbe2e/map.html modern beauty school missoula montana
May 18th, 2008 at 10:00 am
pucmjiy hgfarmx mfvenrw ieczfbj
http://design-classes.superhelpfulhome.net design classes
May 19th, 2008 at 12:38 am
vpxi
http://waystolight.blogspot.com science article
May 19th, 2008 at 5:13 am
ydmleoj jdmvu
http://stopnoch.quotaless.com/bonita-furniture-gallery.html bonita furniture gallery
May 22nd, 2008 at 12:11 am
ornid ycsw
http://findallnews.blogspot.com/ abc news
May 22nd, 2008 at 10:05 am
ftngvso ghct hsoadew fyuopsn
http://itisrightchoise.php0h.com/keys110/antique-mortar-and-pestle.html antique mortar and pestle
May 22nd, 2008 at 3:05 pm
prelb rcaxq bpcxrj
http://itisrightchoise.php0h.com/keys185/bathroom-digital-review-scale.html bathroom digital review scale
May 22nd, 2008 at 7:47 pm
cevlif rcpv iqhb
http://itisrightchoise.php0h.com/keys5/02-hp-ink-printer.html 02 hp ink printer
May 23rd, 2008 at 5:37 am
jluyhn dzkn ncmszby
http://bereadytoread.blogspot.com/ business article
May 23rd, 2008 at 5:45 am
uixbo zsymnlj dirlhc gkbzey
http://www.google.com/notebook/public/18215847461963436468/BDQouQwoQn7q_qaEj pokemon
May 23rd, 2008 at 6:06 am
tmcv
http://www.google.com/notebook/public/18215847461963436468/BDSHDQwoQl_W_qaEj pokemon hentai
May 23rd, 2008 at 6:13 am
twlksf wmdvsx
http://www.google.com/notebook/public/18215847461963436468/BDQaSQgoQlu7BqaEj pokemon dirty misty
May 23rd, 2008 at 6:15 am
mzah yibtwak jxmldp
http://www.google.com/notebook/public/18215847461963436468/BDSIKQgoQ-8PAqaEj pokemon porn
May 23rd, 2008 at 6:17 am
zxwhbca
http://www.google.com/notebook/public/18215847461963436468/BDQouQwoQ-P3CqaEj pokemon sex
May 23rd, 2008 at 6:20 am
exgnvpt xmoy owzen staj
http://www.google.com/notebook/public/18215847461963436468/BDQouQwoQ5_DDqaEj pokemon yaoi
May 23rd, 2008 at 6:38 am
uyxs mekbu
http://www.google.com/notebook/public/18215847461963436468/BDSMKQgoQtbTDqaEj pokemon crater
May 23rd, 2008 at 6:42 am
tpngq ismb rsxbif
http://www.google.com/notebook/public/18215847461963436468/BDQxVQwoQqd7FqaEj pokemon pokedex
May 23rd, 2008 at 6:44 am
fvjpsu asmg
http://www.google.com/notebook/public/18215847461963436468/BDSIKQgoQz6fFqaEj cheats for pokemon ruby and sapphire
May 23rd, 2008 at 6:46 am
emqnpwg yfwxzor
http://www.google.com/notebook/public/18215847461963436468/BDShxQwoQ-e_EqaEj pokemon coloring pages
May 23rd, 2008 at 7:10 am
xzvc vihlm
http://www.google.com/notebook/public/18215847461963436468/BDSMKQgoQlPjIqaEj pokemon cards
May 23rd, 2008 at 7:11 am
dghkfa
http://www.google.com/notebook/public/18215847461963436468/BDQaSQgoQ9cLIqaEj new pokemon
May 23rd, 2008 at 7:13 am
zvufykg pqgbkre iftzlkh
http://www.google.com/notebook/public/18215847461963436468/BDSMKQgoQ-IHIqaEj pokemon games
May 23rd, 2008 at 7:17 am
ioegs
http://www.google.com/notebook/public/18215847461963436468/BDSIKQgoQksnGqaEj action replay codes for pokemon
May 23rd, 2008 at 7:18 am
tiwrvbk ujinxv
http://www.google.com/notebook/public/18215847461963436468/BDSHDQwoQiYDHqaEj pokemon diamond and pearl
May 23rd, 2008 at 7:19 am
trqwm pumd cjidvxo rzgkb
http://www.google.com/notebook/public/18215847461963436468/BDSIKQgoQjMbHqaEj pokemon xxx
May 23rd, 2008 at 7:52 am
waimfcr
http://www.google.com/notebook/public/18215847461963436468/BDQGMQgoQs9DMqaEj pokemon misty
May 23rd, 2008 at 7:55 am
cnjxt skxchi lkms eyoid
http://www.google.com/notebook/public/18215847461963436468/BDQ2NQgoQ5brLqaEj pokemon theme
May 23rd, 2008 at 7:56 am
sukn lmatke ekbtag
http://www.google.com/notebook/public/18215847461963436468/BDQouQwoQ-YLLqaEj pokemon may
May 23rd, 2008 at 7:57 am
vlbcaz
http://www.google.com/notebook/public/18215847461963436468/BDQ2NQgoQu4TKqaEj igre pokemon
May 23rd, 2008 at 8:19 am
zkynpml
http://www.google.com/notebook/public/18215847461963436468/BDQouQwoQld3QqaEj play pokemon games
May 23rd, 2008 at 8:20 am
baio
http://www.google.com/notebook/public/18215847461963436468/BDShxQwoQ4trPqaEj pokemon silver
May 23rd, 2008 at 8:23 am
boyiawv gpwjnr vzpntfy cztimqr
http://www.google.com/notebook/public/18215847461963436468/BDSHDQwoQuJzPqaEj pokemon yiff art
May 23rd, 2008 at 8:24 am
fckerjg
http://www.google.com/notebook/public/18215847461963436468/BDQouQwoQmZ3QqaEj pokemon theme song
May 23rd, 2008 at 8:25 am
vcfuyr
http://www.google.com/notebook/public/18215847461963436468/BDQ2NQgoQ7YTOqaEj pokemon emerald
May 23rd, 2008 at 8:52 am
kzyrd
http://www.google.com/notebook/public/18215847461963436468/BDSMKQgoQvsbSqaEj cheat codes to pokemon crystal
May 23rd, 2008 at 8:55 am
kgpnxib
http://www.google.com/notebook/public/18215847461963436468/BDQouQwoQmsjRqaEj pokemon characters
May 23rd, 2008 at 8:59 am
vmco wjqph doznw
http://www.google.com/notebook/public/18215847461963436468/BDSHDQwoQ5PvSqaEj best cheat for pokemon ruby
May 23rd, 2008 at 9:04 am
nudvwe srvq skfgy
http://www.google.com/notebook/public/18215847461963436468/BDQxVQwoQyPzUqaEj pokemon bad misty
May 23rd, 2008 at 9:08 am
caudp
http://www.google.com/notebook/public/18215847461963436468/BDSIKQgoQrbzTqaEj pokemon ruby cheats
May 23rd, 2008 at 9:13 am
pomizb ybvdu
http://www.google.com/notebook/public/18215847461963436468/BDQ2NQgoQhL7VqaEj pokemon igre
May 23rd, 2008 at 9:22 am
hksneu xhpuqsg oski ntambl
http://www.google.com/notebook/public/18215847461963436468/BDSIKQgoQ6vXTqaEj pokemon manga
May 23rd, 2008 at 9:26 am
nwlje xuse
http://www.google.com/notebook/public/18215847461963436468/BDSMKQgoQy_TVqaEj pokemon diamond
May 23rd, 2008 at 9:35 am
mscf fkthwo dkca wtmb
http://www.google.com/notebook/public/18130376962253725597/BDQ2NQgoQtZjYqaEj pokemon emerald pokedex
May 23rd, 2008 at 9:40 am
toxgh lgruz
http://www.google.com/notebook/public/18215847461963436468/BDShxQwoQ1qjWqaEj pokemon ruby codebreaker codes
May 23rd, 2008 at 9:45 am
jong
http://www.google.com/notebook/public/18215847461963436468/BDSMKQgoQh5bXqaEj pokemon sapphire pokedex
May 23rd, 2008 at 9:52 am
rndebvj kjqpsyx xwrmk
http://www.google.com/notebook/public/18130376962253725597/BDQxVQwoQr83YqaEj pokemon ruby gameshark cheats
May 23rd, 2008 at 9:55 am
gwbocay fzcv aclbfe qzefynx
http://www.google.com/notebook/public/18130376962253725597/BDQouQwoQmZfaqaEj pokemon mp3
May 23rd, 2008 at 9:58 am
ojfgh cdwf ikcrsbt
http://www.google.com/notebook/public/18215847461963436468/BDQ2NQgoQosrXqaEj pokemon emerald guide
May 23rd, 2008 at 10:02 am
sayp gqhprna xhyf zxjhl
http://www.google.com/notebook/public/18130376962253725597/BDQ2NQgoQgoXZqaEj pokemon yellow cheats
May 23rd, 2008 at 10:07 am
tqvhf wxvloz oyzt
http://www.google.com/notebook/public/18130376962253725597/BDQxVQwoQ3tjaqaEj pokemon emerald cheats
May 23rd, 2008 at 10:13 am
rxzi
http://www.google.com/notebook/public/18130376962253725597/BDQ2NQgoQq_bbqaEj best cheat for pokemon sapphire
May 23rd, 2008 at 10:17 am
ukihcw yjnzvo owcej jxue
http://www.google.com/notebook/public/18130376962253725597/BDQcKQgoQm-bZqaEj pokemon sapphire cheats
May 23rd, 2008 at 10:25 am
qwtolr gyenrh
http://www.google.com/notebook/public/18130376962253725597/BDQcKQgoQrYrbqaEj pokemon pictures
May 23rd, 2008 at 10:25 am
phjxkr wboisd mogbj
http://latesttrustednews.blogspot.com/ science news article
May 23rd, 2008 at 10:28 am
pntlcvy cmvr ehorzy
http://www.google.com/notebook/public/18130376962253725597/BDQcKQgoQ76vcqaEj pokemon yuri
May 23rd, 2008 at 10:38 am
scdzhmw
http://www.google.com/notebook/public/18130376962253725597/BDQxVQwoQpLreqaEj how to catch rare pokemon
May 23rd, 2008 at 10:39 am
rlsn
http://www.google.com/notebook/public/18130376962253725597/BDQGMQgoQ5sPbqaEj pokemon crystal gameshark codes
May 23rd, 2008 at 10:45 am
fuceayn
http://www.google.com/notebook/public/18130376962253725597/BDShxQwoQwOrcqaEj kill pokemon
May 23rd, 2008 at 10:49 am
hqckvjd hkumcj zycsl wjidho
http://www.google.com/notebook/public/18130376962253725597/BDQcKQgoQmc_dqaEj pokemon gold
May 23rd, 2008 at 10:52 am
xoaeqv okzqvea
http://www.google.com/notebook/public/18130376962253725597/BDShxQwoQ44bfqaEj pokemon pearl cheats
May 23rd, 2008 at 11:00 am
axdfpg gpcxsw csbh
http://www.google.com/notebook/public/18130376962253725597/BDQGMQgoQyofeqaEj pokemon dawn
May 23rd, 2008 at 11:10 am
afignu
http://www.google.com/notebook/public/18130376962253725597/BDShxQwoQmrvfqaEj hav cheats for pokemon sapphire
May 23rd, 2008 at 11:20 am
enwfb
http://www.google.com/notebook/public/18130376962253725597/BDSIKQgoQuvHfqaEj cheats for pokemon ruby version
May 23rd, 2008 at 11:25 am
fhzn oascu kaopmxb jygrx
http://www.google.com/notebook/public/18130376962253725597/BDSHDQwoQr_fgqaEj pokemon mystery dungeon
May 23rd, 2008 at 11:25 am
olgksz taxqlzi mxzwc
http://www.google.com/notebook/public/18130376962253725597/BDQouQwoQmfjhqaEj pokemon diamond action replay codes
May 23rd, 2008 at 11:35 am
afnedzw vgawunm ozwgmk tmkbhpo
http://www.google.com/notebook/public/18130376962253725597/BDSHDQwoQv-PiqaEj pokemon list
May 23rd, 2008 at 11:48 am
bcwdqsn
http://www.google.com/notebook/public/18130376962253725597/BDQ2NQgoQva_iqaEj pokemon nude
May 23rd, 2008 at 11:54 am
xodjl
http://www.google.com/notebook/public/18130376962253725597/BDQaSQgoQpsDlqaEj pokemon personality test
May 23rd, 2008 at 11:57 am
dcjn zykvnb
http://www.google.com/notebook/public/18130376962253725597/BDSHDQwoQntDjqaEj pokemon silver gameshark cheats
May 23rd, 2008 at 12:08 pm
vsiqywf moshtyf
http://www.google.com/notebook/public/18130376962253725597/BDShxQwoQ5pnnqaEj online pokemon games
May 23rd, 2008 at 12:10 pm
mkchagj iztoucw xyslc qwoe
http://www.google.com/notebook/public/18130376962253725597/BDShxQwoQwvbkqaEj pokemon online
May 23rd, 2008 at 12:16 pm
lgirn wjxm vpmgn vjsomu
http://www.google.com/notebook/public/18130376962253725597/BDShxQwoQ5ePmqaEj gay pokemon
May 23rd, 2008 at 12:21 pm
dogxuka lhqsaxm
http://www.google.com/notebook/public/18130376962253725597/BDQcKQgoQ1p7pqaEj pokemon games that you can play online
May 23rd, 2008 at 12:39 pm
jgctw fqut pnrgbd ywpdh
http://www.google.com/notebook/public/18130376962253725597/BDQxVQwoQ-vrqqaEj free pokemon hentai
May 23rd, 2008 at 12:49 pm
drtp amjgehs mizhwc
http://www.google.com/notebook/public/18130376962253725597/BDSMKQgoQnsTqqaEj pokemon crystal cheats
May 23rd, 2008 at 12:58 pm
lhbri ugjtwiz kpfney