Sponsored Links:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[RELEASE] NaN Scrapers
#1
Last week was the official release of the NoobsandNerds Scrapers database. Before I go into the purpose of this database I just want to quickly explain for those who do not know, what a scraper actually is..

Scrapers are what alot of addons use to access the direct links to Movies and TV Shows from websites. Some addons that use Scrapers are Specto, Zen and most recently Bob.

Currently alot of addons that scrape are using the scrapers which were coded by Lambda for Genesis and now Exodus. Which means when they break they have to wait for Lambda to fix them himself.

NaN Scrapers has been made in the hope that the whole community can benefit from and help contribute to it. The idea is that anyone in the Community can contibute to helping add to or fix scrapers in the NaN Scraper Library and in turn new or current Addon Developers can use the scrapers from this library in their own addons.

One addon already benefiting from this is the Bob playlist addon and there is also another Developer already in the proccess of converting their addon to work with NaN Scrapers.

Using the NaN Scraper library will hopefully help new devs when creating addons by taking away the time consuming proccess of coding up scrapers. They can then get their addon out there and help the rest of the Developers in the community by contributing to the NaN Scraper library.

As said above this project has been created in the hope Developers come together to help eachother by fixing and adding new scrapers to the Library which they can then use in their addons. The scrapers have been coded in a very easy to follow format so it should be easy for anyone familiar with code to create their own or help with any issues with the scraper.

If your an addon Developer and are interested in being part of this project feel free to get in touch.
[Image: nansig_arb.png]
  Reply
Sponsored Links:
#2
Would maybe need pointing what to put where but think i could help
  Reply
#3
(11-05-2016, 08:52 PM)toall1985 Wrote: Would maybe need pointing what to put where but think i could help
That would be great. Do you have Telegram? We have a scraper group in there I will add you to.

Sent from my SM-N910F using Tapatalk
[Image: nansig_arb.png]
  Reply
#4
(11-05-2016, 09:26 PM)andrewrb Wrote:
(11-05-2016, 08:52 PM)toall1985 Wrote: Would maybe need pointing what to put where but think i could help
That would be great. Do you have Telegram? We have a scraper group in there I will add you to.

Sent from my SM-N910F using Tapatalk

i do, give me 5 and ill try remember my details and message you them
  Reply
#5
Is there a post/link to the required format, what needs to be returned and how?
  Reply
#6
not yet, i'll try and make a post about it here tonight. but if you realy want to know now you can check the code. all scrapers inherit from a common base class and the output format is the same for all of them, so if you open one you can see it very easily
  Reply
#7
so as a user of the library there are 6 methods you are likely to call:

  • Code:
    scrape_movie(title, year, imdb, host=None, include_disabled=False, timeout=30)
    this scrapes for a movie. imdb can be None, but some scrapers need it so you'll get less results. gives back a ThreadPoolExecutor object which when called returns arrays of dicts of the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}
  • Code:
    scrape_movie_with_dialog(title, year, imdb, host=None, include_disabled=False, timeout=30)
    this scrapes for a movie. imdb can be None, but some scrapers need it so you'll get less results. pops up a dialog that autopopulates as links come in from the scrapers (close to how metalliq player selection   window works). returns a dict in the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}
  • Code:
    scrape_episode(title, show_year, year, season, episode, imdb, tvdb, host=None, include_disabled=False, timeout=30)
    this scrapes for an episode. imdb and tvdb can be None, but some scrapers need it so you'll get less results. gives back a ThreadPoolExecutor object which when called returns arrays of dicts of the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}
  • Code:
    scrape_episode_with_dialog(title, show_year, year, season, episode, imdb, tvdb, host=None, include_disabled=False, timeout=30)
    this scrapes for an episode. imdb and tvdb can be None, but some scrapers need it so you'll get less results. pops up a dialog that autopopulates as links come in from the scrapers (close to how metalliq player selection   window works). returns a dict in the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}
  • Code:
    scrape_song(title, artist, host=None, include_disabled=False, timeout=30)
    this scrapes for a song. gives back a ThreadPoolExecutor object which when called returns arrays of dicts of the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}
  • Code:
    scrape_song_with_dialog(title, artist, host=None, include_disabled=False, timeout=30)
    this scrapes for a song. pops up a dialog that autopopulates as links come in from the scrapers (close to how metalliq player selection   window works). returns a dict in the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}

as a developer writing scrapers you do it by inheriting from the scraper class located in nanscrapers.scraper and overwriting any of the following methods:
  • Code:
    scrape_movie(self, title, year, imdb)
    needs to return an array of dicts of the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}
  • Code:
    scrape_episode(self,title, show_year, year, season, episode, imdb, tvdb)
    needs to return an array of dicts of the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}
  • Code:
    scrape_music(self, title, artist)
    needs to return an array of dicts of the following format:
    Code:
    {'source': string, 'quality': string, 'scraper': string, 'url': string, 'direct': Boolean}
  • Code:
    get_settings_xml(clas)
    needs to return an array of strings to be used in settings.xml
  Reply
#8
Hey, thanks to the development video series, I *finally* got into Kodi addon development after years of thinking about it. Where can I get this scraper to use and contribute to it?

Thanks!
  Reply
#9
(03-05-2017, 11:19 AM)TechnicalFoul Wrote: Hey, thanks to the development video series, I *finally* got into Kodi addon development after years of thinking about it. Where can I get this scraper to use and contribute to it?

Thanks!
Hi. You can find it here.

https://github.com/noobsandnerds/noobsan...anscrapers

Sent from my SM-N910F using Tapatalk
[Image: nansig_arb.png]
  Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)