Max/SolidScribe
1
0
Fork 0
Code Issues 24 Pull Requests Releases Wiki Activity

Web Scrape Improvements #33

New Issue
Open
opened 2020-03-20 04:34:05 -07:00 by Max · 1 comment
Max commented 2020-03-20 04:34:05 -07:00
Owner
Copy Link
Copy Source
  • Better Error catching on scrape
  • Default to favicon if image can't be found
  • Scrape large sections of text
    ** Try to isolate article bodies
    ** Try to isolate recipe, targeting keywords like tablespoon, tbsp, etc
* Better Error catching on scrape * Default to favicon if image can't be found * Scrape large sections of text ** Try to isolate article bodies ** Try to isolate recipe, targeting keywords like tablespoon, tbsp, etc
Max added the Enhancement label 2020-03-20 04:34:05 -07:00
Max changed title from Web Scrap Improvements to Web Scrape Improvements 2020-03-20 04:42:21 -07:00
Max commented 2020-04-09 15:45:16 -07:00
Author
Owner
Copy Link
Copy Source

Scrape Improvement Notes

  • Seperate out bodies of logic that scrape certain sections

    • Keyword scrape logic
    • Image scrape logic
    • Text content article/recipe content

  • Only prepend URL to images that don't have a full URL

  • Don't try to process ico files

  • If a small image is scraped, don't try to resize it

  • Don't pass URL params to scrape like ?v=htu

Scrape Improvement Notes * Seperate out bodies of logic that scrape certain sections * * Keyword scrape logic * * Image scrape logic * * Text content article/recipe content * Only prepend URL to images that don't have a full URL * Don't try to process ico files * If a small image is scraped, don't try to resize it * Don't pass URL params to scrape like ?v=htu
Max added the In Progress label 2020-04-09 15:45:23 -07:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Bug
An issue to fix
 Enhancement
Update or improvement to existing implementation.
 Idea
Potential features that can be considered.
 In Progress
I'm Currently working on this feature
 New Feature
New Features
 Security
Updates to application and server security
No labels Enhancement In Progress
Milestone
No items
No Milestone
Assignees
Clear assignees
Max (Max Gialanella)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Max/SolidScribe#33