Max/SolidScribe
Max/SolidScribe
1
0
Fork 0
Code Issues 24 Pull Requests Releases Wiki Activity

Web Scrape Improvements #33

New Issue
Open
opened 2020-03-20 04:34:05 -07:00 by Max · 1 comment
Max commented 2020-03-20 04:34:05 -07:00
Owner
Copy Link
  • Better Error catching on scrape
  • Default to favicon if image can't be found
  • Scrape large sections of text
    ** Try to isolate article bodies
    ** Try to isolate recipe, targeting keywords like tablespoon, tbsp, etc
* Better Error catching on scrape * Default to favicon if image can't be found * Scrape large sections of text ** Try to isolate article bodies ** Try to isolate recipe, targeting keywords like tablespoon, tbsp, etc
Max added the
Enhancement
 label 2020-03-20 04:34:05 -07:00
Max changed title from Web Scrap Improvements to Web Scrape Improvements 2020-03-20 04:42:21 -07:00
Max commented 2020-04-09 15:45:16 -07:00
Author
Owner
Copy Link

Scrape Improvement Notes

  • Seperate out bodies of logic that scrape certain sections

    • Keyword scrape logic
    • Image scrape logic
    • Text content article/recipe content

  • Only prepend URL to images that don't have a full URL

  • Don't try to process ico files

  • If a small image is scraped, don't try to resize it

  • Don't pass URL params to scrape like ?v=htu

Scrape Improvement Notes * Seperate out bodies of logic that scrape certain sections * * Keyword scrape logic * * Image scrape logic * * Text content article/recipe content * Only prepend URL to images that don't have a full URL * Don't try to process ico files * If a small image is scraped, don't try to resize it * Don't pass URL params to scrape like ?v=htu
Max added the
In Progress
 label 2020-04-09 15:45:23 -07:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Bug
An issue to fix
Enhancement
Update or improvement to existing implementation.
Idea
Potential features that can be considered.
In Progress
I'm Currently working on this feature
New Feature
New Features
Security
Updates to application and server security
No Label
Enhancement
In Progress
Milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Max/SolidScribe#33
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?