Refactoring and cleaning up

Author: jmerle

docs/Scraper-Reference.md

@@ -187,39 +187,41 @@ More information about how filters work is available on the [Filter Reference](.
 
 ## Keeping scrapers up-to-date
 
-In order to keep scrapers up-to-date the `get_latest_version(options, &block)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations.
+In order to keep scrapers up-to-date the `get_latest_version(opts)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations.
 
 To make life easier, there are a few utility methods that you can use in `get_latest_version`:
-* `fetch(url, options, &block)`
+* `fetch(url, opts)`
 
-  Makes a GET request to the url and calls `&block` with the body.
+  Makes a GET request to the url and returns the response body.
 
   Example: [lib/docs/scrapers/bash.rb](../lib/docs/scrapers/bash.rb)
-* `fetch_doc(url, options, &block)`
+* `fetch_doc(url, opts)`
 
-  Makes a GET request to the url and calls `&block` with the HTML body converted to a Nokogiri document.
+  Makes a GET request to the url and returns the HTML body converted to a Nokogiri document.
 
   Example: [lib/docs/scrapers/git.rb](../lib/docs/scrapers/git.rb)
-* `fetch_json(url, options, &block)`
+* `fetch_json(url, opts)`
 
-  Makes a GET request to the url and calls `&block` with the JSON body converted to a dictionary.
-* `get_npm_version(package, options, &block)`
+  Makes a GET request to the url and returns the JSON body converted to a dictionary.
 
-  Calls `&block` with the latest version of the given npm package.
+  Example: [lib/docs/scrapers/mdn/mdn.rb](../lib/docs/scrapers/mdn/mdn.rb)
+* `get_npm_version(package, opts)`
+
+  Returns the latest version of the given npm package.
 
   Example: [lib/docs/scrapers/bower.rb](../lib/docs/scrapers/bower.rb)
-* `get_latest_github_release(owner, repo, options, &block)`
+* `get_latest_github_release(owner, repo, opts)`
 
-  Calls `&block` with the latest GitHub release of the given repository ([format](https://developer.github.com/v3/repos/releases/#get-the-latest-release)).
+  Returns the latest GitHub release of the given repository ([format](https://developer.github.com/v3/repos/releases/#get-the-latest-release)).
 
   Example: [lib/docs/scrapers/jsdoc.rb](../lib/docs/scrapers/jsdoc.rb)
-* `get_github_tags(owner, repo, options, &block)`
+* `get_github_tags(owner, repo, opts)`
 
-  Calls `&block` with the list of tags on the given repository ([format](https://developer.github.com/v3/repos/#list-tags)).
+  Returns the list of tags on the given repository ([format](https://developer.github.com/v3/repos/#list-tags)).
 
   Example: [lib/docs/scrapers/liquid.rb](../lib/docs/scrapers/liquid.rb)
-* `get_github_file_contents(owner, repo, path, options, &block)`
+* `get_github_file_contents(owner, repo, path, opts)`
 
-  Calls `&block` with the contents of the requested file in the default branch of the given repository.
+  Returns the contents of the requested file in the default branch of the given repository.
 
   Example: [lib/docs/scrapers/minitest.rb](../lib/docs/scrapers/minitest.rb)

lib/docs/core/doc.rb

@@ -164,24 +164,23 @@ def build_pages(&block)
       raise NotImplementedError
     end
 
-    def get_scraper_version(opts, &block)
+    def get_scraper_version(opts)
       if self.class.method_defined?(:options) and !options[:release].nil?
-        block.call options[:release]
+        options[:release]
       else
         # If options[:release] does not exist, we return the Epoch timestamp of when the doc was last modified in DevDocs production
-        fetch_json('https://devdocs.io/docs.json', opts) do |json|
-          items = json.select {|item| item['name'] == self.class.name}
-          items = items.map {|item| item['mtime']}
-          block.call items.max
-        end
+        json = fetch_json('https://devdocs.io/docs.json', opts)
+        items = json.select {|item| item['name'] == self.class.name}
+        items = items.map {|item| item['mtime']}
+        items.max
       end
     end
 
     # Should return the latest version of this documentation
     # If options[:release] is defined, it should be in the same format
     # If options[:release] is not defined, it should return the Epoch timestamp of when the documentation was last updated
     # If the docs will never change, simply return '1.0.0'
-    def get_latest_version(options, &block)
+    def get_latest_version(opts)
       raise NotImplementedError
     end
 
@@ -216,55 +215,49 @@ def is_outdated(scraper_version, latest_version)
     # Utility methods for get_latest_version
     #
 
-    def fetch(url, options, &block)
+    def fetch(url, opts)
       headers = {}
 
-      if options.key?(:github_token) and url.start_with?('https://api.github.com/')
-        headers['Authorization'] = "token #{options[:github_token]}"
+      if opts.key?(:github_token) and url.start_with?('https://api.github.com/')
+        headers['Authorization'] = "token #{opts[:github_token]}"
       end
 
-      options[:logger].debug("Fetching #{url}")
+      opts[:logger].debug("Fetching #{url}")
+      response = Request.run(url, { headers: headers })
 
-      Request.run(url, { headers: headers }) do |response|
-        if response.success?
-          block.call response.body
-        else
-          options[:logger].error("Couldn't fetch #{url} (response code #{response.code})")
-          block.call nil
-        end
+      if response.success?
+        response.body
+      else
+        opts[:logger].error("Couldn't fetch #{url} (response code #{response.code})")
+        nil
       end
     end
 
-    def fetch_doc(url, options, &block)
-      fetch(url, options) do |body|
-        block.call Nokogiri::HTML.parse(body, nil, 'UTF-8')
-      end
+    def fetch_doc(url, opts)
+      body = fetch(url, opts)
+      Nokogiri::HTML.parse(body, nil, 'UTF-8')
     end
 
-    def fetch_json(url, options, &block)
-      fetch(url, options) do |body|
-        block.call JSON.parse(body)
-      end
+    def fetch_json(url, opts)
+      JSON.parse fetch(url, opts)
     end
 
-    def get_npm_version(package, options, &block)
-      fetch_json("https://registry.npmjs.com/#{package}", options) do |json|
-        block.call json['dist-tags']['latest']
-      end
+    def get_npm_version(package, opts)
+      json = fetch_json("https://registry.npmjs.com/#{package}", opts)
+      json['dist-tags']['latest']
     end
 
-    def get_latest_github_release(owner, repo, options, &block)
-      fetch_json("https://api.github.com/repos/#{owner}/#{repo}/releases/latest", options, &block)
+    def get_latest_github_release(owner, repo, opts)
+      fetch_json("https://api.github.com/repos/#{owner}/#{repo}/releases/latest", opts)
     end
 
-    def get_github_tags(owner, repo, options, &block)
-      fetch_json("https://api.github.com/repos/#{owner}/#{repo}/tags", options, &block)
+    def get_github_tags(owner, repo, opts)
+      fetch_json("https://api.github.com/repos/#{owner}/#{repo}/tags", opts)
     end
 
-    def get_github_file_contents(owner, repo, path, options, &block)
-      fetch_json("https://api.github.com/repos/#{owner}/#{repo}/contents/#{path}", options) do |json|
-        block.call(Base64.decode64(json['content']))
-      end
+    def get_github_file_contents(owner, repo, path, opts)
+      json = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/contents/#{path}", opts)
+      Base64.decode64(json['content'])
     end
   end
 end

lib/docs/scrapers/angular.rb

@@ -155,8 +155,8 @@ def handle_response(response)
       end
     end
 
-    def get_latest_version(options, &block)
-      get_npm_version('@angular/core', options, &block)
+    def get_latest_version(opts)
+      get_npm_version('@angular/core', opts)
     end
 
     private

lib/docs/scrapers/angularjs.rb

@@ -70,8 +70,8 @@ class Angularjs < UrlScraper
       self.base_url = "https://code.angularjs.org/#{release}/docs/partials/"
     end
 
-    def get_latest_version(options, &block)
-      get_npm_version('angular', options, &block)
+    def get_latest_version(opts)
+      get_npm_version('angular', opts)
     end
   end
 end

lib/docs/scrapers/ansible.rb

@@ -88,10 +88,9 @@ class Ansible < UrlScraper
         list_of_all_modules.html)
     end
 
-    def get_latest_version(options, &block)
-      fetch_doc('https://docs.ansible.com/ansible/latest/index.html', options) do |doc|
-        block.call doc.at_css('.DocSiteProduct-CurrentVersion').content.strip
-      end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.ansible.com/ansible/latest/index.html', opts)
+      doc.at_css('.DocSiteProduct-CurrentVersion').content.strip
     end
   end
 end

lib/docs/scrapers/apache.rb

@@ -34,10 +34,9 @@ class Apache < UrlScraper
       Licensed under the Apache License, Version 2.0.
     HTML
 
-    def get_latest_version(options, &block)
-      fetch_doc('http://httpd.apache.org/docs/', options) do |doc|
-        block.call doc.at_css('#apcontents > ul a')['href'][0...-1]
-      end
+    def get_latest_version(opts)
+      doc = fetch_doc('http://httpd.apache.org/docs/', opts)
+      doc.at_css('#apcontents > ul a')['href'][0...-1]
     end
   end
 end

lib/docs/scrapers/apache_pig.rb

@@ -43,11 +43,10 @@ class ApachePig < UrlScraper
       self.base_url = "https://pig.apache.org/docs/r#{release}/"
     end
 
-    def get_latest_version(options, &block)
-      fetch_doc('https://pig.apache.org/', options) do |doc|
-        item = doc.at_css('div[id="menu_1.2"] > .menuitem:last-child')
-        block.call item.content.strip.sub(/Release /, '')
-      end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://pig.apache.org/', opts)
+      item = doc.at_css('div[id="menu_1.2"] > .menuitem:last-child')
+      item.content.strip.sub(/Release /, '')
     end
   end
 end

lib/docs/scrapers/async.rb

@@ -18,11 +18,9 @@ class Async < UrlScraper
       Licensed under the MIT License.
     HTML
 
-    def get_latest_version(options, &block)
-      fetch_doc('https://caolan.github.io/async/', options) do |doc|
-        version = doc.at_css('#version-dropdown > a').content.strip[1..-1]
-        block.call version
-      end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://caolan.github.io/async/', opts)
+      doc.at_css('#version-dropdown > a').content.strip[1..-1]
     end
   end
 end

lib/docs/scrapers/babel.rb

@@ -23,10 +23,9 @@ class Babel < UrlScraper
       '<div></div>'
     end
 
-    def get_latest_version(options, &block)
-      fetch_doc('https://babeljs.io/docs/en/', options) do |doc|
-        block.call doc.at_css('a[href="/versions"] > h3').content
-      end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://babeljs.io/docs/en/', opts)
+      doc.at_css('a[href="/versions"] > h3').content
     end
   end
 end

lib/docs/scrapers/backbone.rb

@@ -21,11 +21,9 @@ class Backbone < UrlScraper
       Licensed under the MIT License.
     HTML
 
-    def get_latest_version(options, &block)
-      fetch_doc('https://backbonejs.org/', options) do |doc|
-        version = doc.at_css('.version').content
-        block.call version[1...-1]
-      end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://backbonejs.org/', opts)
+      doc.at_css('.version').content[1...-1]
     end
   end
 end