diff --git a/client/index.html b/client/index.html index 70a68fa..b01d7bc 100644 --- a/client/index.html +++ b/client/index.html @@ -10,10 +10,23 @@ - Notes + Solid Scribe - A Note Taking Website -
+
+ +

You have found a Solid Scribe

+ logo +

Solid Scribe

+

A note application that respects your privacy.

+

Take notes with a clean editor that works on desktop or mobile.

+

Search notes, links and files to find what you need.

+

Accessable everywhere.

+

Categorize notes with tags.

+

Share data with fellow users.

+

Encrypt notes for additional security.

+ This site requires Javascipt to run. +
diff --git a/client/src/pages/HelpPage.vue b/client/src/pages/HelpPage.vue index 16d75cf..2b27ba1 100644 --- a/client/src/pages/HelpPage.vue +++ b/client/src/pages/HelpPage.vue @@ -2,17 +2,16 @@
+
+

Help

-
-
-
-

Quick Note

The Quick note feature was designed to allow rapid input to a single note. Rather than junking up all your notes with random links, numbers or haikus, you can put them all in one place.

All data pushed to the quick note can still be edited like a normal note.


Dark Theme

Dark theme was designed to minimize the amount of blue. Less blue entering your eyes is supposed to help you fall asleep.

Most things turn sepia and a filter is applied to images to make them more sepia.

Here is some good research on the topic: https://justgetflux.com/research.html


Password Protected Notes

Note protected with a password are encrypted. This means the data is scrambled and unreadable unless the correct password is used to decrypt them.

If a password is forgotten, it can never be recovered. Passwords are not saved for encrypted notes. If you lose the password to a protected note, that note text is lost.

Only the text of the note is protected. Tags, Files attached to the note, and the title of the note are still visible without a password. You can not search text in a password protected note. But you can search by the title.


Links in notes

Links put into notes are automatically scraped. This means the data from the link will be scanned to get an image and some text from the website to help make that link more accessible in the future.


Files in notes

Files can be uploaded to notes. If its an image, the picture will be put into the note.

Images added to notes will have the text pulled out so it can be searched (This isn't super accurate so don't rely to heavily on it.) The text can be updated at any time.


Deleting notes

When notes are deleted, none of the files related to the note are deleted.

+

Quick Note

The Quick note feature was designed to allow rapid input to a single note. Rather than junking up all your notes with random links, numbers or haikus, you can put them all in one place.

All data pushed to the quick note can still be edited like a normal note.


Dark Theme

Dark theme was designed to minimize the amount of blue. Less blue entering your eyes is supposed to help you fall asleep.

Most things turn sepia and a filter is applied to images to make them more sepia.

Here is some good research on the topic: https://justgetflux.com/research.html


Password Protected Notes

Note protected with a password are encrypted. This means the data is scrambled and unreadable unless the correct password is used to decrypt them.

If a password is forgotten, it can never be recovered. Passwords are not saved for encrypted notes. If you lose the password to a protected note, that note text is lost.

Only the text of the note is protected. Tags, Files attached to the note, and the title of the note are still visible without a password. You can not search text in a password protected note. But you can search by the title.


Links in notes

Links put into notes are automatically scraped. This means the data from the link will be scanned to get an image and some text from the website to help make that link more accessible in the future.


Files in notes

Files can be uploaded to notes. If its an image, the picture will be put into the note.

Images added to notes will have the text pulled out so it can be searched (This isn't super accurate so don't rely to heavily on it.) The text can be updated at any time.


Deleting notes

When notes are deleted, none of the files related to the note are deleted.


Daily Backups

All notes are backed up, every night, at midnight. If there is data loss, it can be restored from a backup. If you experience some sort of cataclysmic data loss please contact the system administrator for a copy of your data or restoration a restoration procedure.

diff --git a/server/helpers/SiteScrape.js b/server/helpers/SiteScrape.js new file mode 100644 index 0000000..39472a4 --- /dev/null +++ b/server/helpers/SiteScrape.js @@ -0,0 +1,151 @@ +let SiteScrape = module.exports = {} + +// +// $ = the cheerio scrape object +// + +const removeWhitespace = /\s+/g + +const commonWords = ['share','facebook','twitter','reddit','be','have','do','say','get','make','go','know','take','see','come','think','look','want', + 'give','use','find','tell','ask','work','seem','feel','try','leave','call','good','new','first','last','long','great','little','own','other','old', + 'right','big','high','different','small','large','next','early','young','important','few','public','bad','same','able','to','of','in','for','on', + 'with','at','by','from','up','about','into','over','after','the','and','a','that','I','it','not','he','as','you','this','but','his','they','her', + 'she','or','an','will','my','one','all','would','there','their','and','that','but','or','as','if','when','than','because','while','where','after', + 'so','though','since','until','whether','before','although','nor','like','once','unless','now','except','are','also','is','your','its'] + +SiteScrape.getTitle = ($) => { + + let title = $('title').text().replace(removeWhitespace, " ") + return title + +} + +//Finds all urls in text, removes duplicates, makes sure they have https:// +SiteScrape.getCleanUrls = (textBlock) => { + //Find all URLs in text + //@TODO - Use the process text library for this function + const urlPattern = /(?:(?:https?|ftp|file):\/\/|www\.|ftp\.)(?:\([-A-Z0-9+&@#/%=~_|$?!:,.]*\)|[-A-Z0-9+&@#/%=~_|$?!:,.])*(?:\([-A-Z0-9+&@#/%=~_|$?!:,.]*\)|[A-Z0-9+&@#/%=~_|$])/igm + let allUrls = textBlock.match(urlPattern) + + if(allUrls == null){ + return [] + } + + //Every URL needs HTTPS!!! + let foundUrls = [] + allUrls.forEach( (item, index) => { + //add protocol if it is missing + if(item.indexOf('https://') == -1 && item.indexOf('http://') == -1){ + allUrls[index] = 'https://'+item + } + //convert http to https + if(item.indexOf('http://') >= 0){ + allUrls[index] = item.replace('http://','https://') + } + }) + + //Remove all duplicates + foundUrls = [...new Set(allUrls)] + + return foundUrls +} + +//Site hostname with https:// eg: https://www.google.com +SiteScrape.getHostName = (url) => { + + var hostname = 'https://'+(new URL(url)).hostname; + console.log('hostname', hostname) + return hostname +} + +// URL for image that can be downloaded to represent website +SiteScrape.getDisplayImage = ($, url) => { + + const hostname = SiteScrape.getHostName(url) + + let metaImg = $('meta[property="og:image"]') + let shortcutIcon = $('link[rel="shortcut icon"]') + let favicon = $('link[rel="icon"]') + let randomImg = $('img') + + console.log('----') + + //Scrape metadata for page image + //Grab the first random image we find + if(randomImg && randomImg[0] && randomImg[0].attribs){ + thumbnail = hostname + randomImg[0].attribs.src + console.log('random img '+thumbnail) + } + //Grab the favicon of the site + if(favicon && favicon[0] && favicon[0].attribs){ + thumbnail = hostname + favicon[0].attribs.href + console.log('favicon '+thumbnail) + } + //Grab the shortcut icon + if(shortcutIcon && shortcutIcon[0] && shortcutIcon[0].attribs){ + thumbnail = hostname + shortcutIcon[0].attribs.href + console.log('shortcut '+thumbnail) + } + //Grab the presentation image for the site + if(metaImg && metaImg[0] && metaImg[0].attribs){ + thumbnail = metaImg[0].attribs.content + console.log('ogImg '+thumbnail) + } + + console.log('-----') + return thumbnail +} + +// Get all the site text and parse out the words that appear most +SiteScrape.getKeywords = ($) => { + + let majorContent = '' + + majorContent += $('[class*=content]').text() + .replace(removeWhitespace, " ") //Remove all whitespace + .replace(/\W\s/g, '') //Remove all non alphanumeric characters + .substring(0,3000) //Limit to 3000 characters + .toLowerCase() + + //Count frequency of each word in scraped text + let frequency = {} + majorContent.split(' ').forEach(word => { + if(commonWords.includes(word)){ + return //Exclude certain words + } + if(!frequency[word]){ + frequency[word] = 0 + } + frequency[word]++ + }) + + //Create a sortable array + var sortable = []; + for (var index in frequency) { + if(frequency[index] > 1){ + sortable.push([index, frequency[index]]); + } + } + + //Sort them by most used words in the list + sortable.sort(function(a, b) { + return b[1] - a[1]; + }); + + let finalWords = [] + for(let i=0; i<5; i++){ + if(sortable[i] && sortable[i][0]){ + finalWords.push(sortable[i][0]) + } + } + + if(finalWords.length > 0){ + return 'Keywords: ' + finalWords.join(', ') + } + return '' +} + +SiteScrape.getMainText = ($) => {} + + + diff --git a/server/index.js b/server/index.js index 23b09e8..9ca8deb 100644 --- a/server/index.js +++ b/server/index.js @@ -128,6 +128,18 @@ app.use(function(req, res, next){ } }) +// Testing Area +// let att = require('@models/Attachment') +// let testUrl = 'https://dba.stackexchange.com/questions/23908/how-to-search-a-mysql-database-with-encrypted-fields' +// testUrl = 'https://www.solidscribe.com/#/' +// console.log('About to scrape: ', testUrl) +// att.processUrl(61, 3213, testUrl) +// .then(results => { +// console.log('Scrape happened') +// }) +// +// + //Test app.get(prefix, (req, res) => res.send('The api is running')) diff --git a/server/models/Attachment.js b/server/models/Attachment.js index 2529868..0699839 100644 --- a/server/models/Attachment.js +++ b/server/models/Attachment.js @@ -1,5 +1,7 @@ let db = require('@config/database') +let SiteScrape = require('@helpers/SiteScrape') + let Attachment = module.exports = {} const cheerio = require('cheerio') @@ -242,32 +244,8 @@ Attachment.scanTextForWebsites = (io, userId, noteId, noteText) => { Attachment.urlForNote(userId, noteId).then(attachments => { - //Find all URLs in text - //@TODO - Use the process text library for this function - const urlPattern = /(?:(?:https?|ftp|file):\/\/|www\.|ftp\.)(?:\([-A-Z0-9+&@#/%=~_|$?!:,.]*\)|[-A-Z0-9+&@#/%=~_|$?!:,.])*(?:\([-A-Z0-9+&@#/%=~_|$?!:,.]*\)|[A-Z0-9+&@#/%=~_|$])/igm - let allUrls = noteText.match(urlPattern) - - if(allUrls == null){ - allUrls = [] - } - - //Every URL needs HTTPS!!! - let foundUrls = [] - allUrls.forEach( (item, index) => { - //Every URL should have HTTPS - if(item.indexOf('https://') == -1 && item.indexOf('http://') == -1){ - allUrls[index] = 'https://'+item - } - //URLs should all have HTTPS!!! - if(item.indexOf('http://') >= 0){ - allUrls[index] = item.replace('http://','https://') - } - }) - - //Remove all duplicates - foundUrls = [...new Set(allUrls)] - - + //Pull all the URLs out of the text + let foundUrls = SiteScrape.getCleanUrls(noteText) //Go through each saved URL, remove new URLs from saved URLs //If a URL is not found, delete it @@ -386,14 +364,6 @@ Attachment.processUrl = (userId, noteId, url) => { return new Promise((resolve, reject) => { - const excludeWords = ['share','facebook','twitter','reddit','be','have','do','say','get','make','go','know','take','see','come','think','look','want', - 'give','use','find','tell','ask','work','seem','feel','try','leave','call','good','new','first','last','long','great','little','own','other','old', - 'right','big','high','different','small','large','next','early','young','important','few','public','bad','same','able','to','of','in','for','on', - 'with','at','by','from','up','about','into','over','after','the','and','a','that','I','it','not','he','as','you','this','but','his','they','her', - 'she','or','an','will','my','one','all','would','there','their','and','that','but','or','as','if','when','than','because','while','where','after', - 'so','though','since','until','whether','before','although','nor','like','once','unless','now','except','are','also','is','your','its'] - - var removeWhitespace = /\s+/g const options = { uri: url, @@ -428,70 +398,33 @@ Attachment.processUrl = (userId, noteId, url) => { }) .then($ => { + //Clear timeout that would end this function clearTimeout(requestTimeout) - var desiredSearchText = '' - - let pageTitle = $('title').text().replace(removeWhitespace, " ") - desiredSearchText += pageTitle + "\n" - // let header = $('h1').text().replace(removeWhitespace, " ") // desiredSearchText += header + "\n" - //Scrape metadata for page image - let metadata = $('meta[property="og:image"]') - if(metadata && metadata[0] && metadata[0].attribs){ - thumbnail = metadata[0].attribs.content - } + const pageTitle = SiteScrape.getTitle($) + const hostname = SiteScrape.getHostName(url) - let majorContent = '' - majorContent += $('[class*=content]').text() - .replace(removeWhitespace, " ") //Remove all whitespace - .replace(/\W\s/g, '') //Remove all non alphanumeric characters - .substring(0,3000) - .toLowerCase() - majorContent += $('[id*=content]').text().replace(removeWhitespace, " ") - .replace(removeWhitespace, " ") //Remove all whitespace - .replace(/\W\s/g, '') //Remove all non alphanumeric characters - .substring(0,3000) //Limit characters - .toLowerCase() + const thumbnail = SiteScrape.getDisplayImage($, url) - //Count frequency of each word in scraped text - let frequency = {} - majorContent.split(' ').forEach(word => { - if(excludeWords.includes(word)){ - return //Exclude certain words - } - if(!frequency[word]){ - frequency[word] = 0 - } - frequency[word]++ + const keywords = SiteScrape.getKeywords($) + + var desiredSearchText = '' + desiredSearchText += pageTitle + "\n" + desiredSearchText += keywords + + console.log({ + pageTitle, + hostname, + thumbnail, + keywords }) + - //Create a sortable array - var sortable = []; - for (var index in frequency) { - if(frequency[index] > 1){ - sortable.push([index, frequency[index]]); - } - } - - //Sort them by most used words in the list - sortable.sort(function(a, b) { - return b[1] - a[1]; - }); - - let finalWords = [] - for(let i=0; i<5; i++){ - if(sortable[i] && sortable[i][0]){ - finalWords.push(sortable[i][0]) - } - } - - if(finalWords.length > 0){ - desiredSearchText += 'Keywords: ' + finalWords.join(', ') - } + // throw new Error('Ending this function early.') // console.log('TexT Scraped') @@ -532,9 +465,10 @@ Attachment.processUrl = (userId, noteId, url) => { }) .catch(error => { - console.log('Issue with scrape') + // console.log('Scrape pooped out') + // console.log('Issue with scrape') console.log(error) - resolve('') + // resolve('') }) requestTimeout = setTimeout( () => { diff --git a/server/models/Note.js b/server/models/Note.js index 3a8d69a..9753ed6 100644 --- a/server/models/Note.js +++ b/server/models/Note.js @@ -167,10 +167,10 @@ Note.update = (io, userId, noteId, noteText, noteTitle, color, pinned, archived, return new Promise((resolve, reject) => { //Prevent note loss if it saves with empty text - if(ProcessText.removeHtml(noteText) == ''){ + //if(ProcessText.removeHtml(noteText) == ''){ // console.log('Not saving empty note') // resolve(false) - } + //} const now = Math.round((+new Date)/1000) diff --git a/server/routes/attachmentController.js b/server/routes/attachmentController.js index ddbaf23..8f73dcc 100644 --- a/server/routes/attachmentController.js +++ b/server/routes/attachmentController.js @@ -4,7 +4,7 @@ var multer = require('multer') var upload = multer({ dest: '../staticFiles/' }) //@TODO make this a global value let router = express.Router() -let Attachment = require('@models/Attachment'); +let Attachment = require('@models/Attachment') let Note = require('@models/Note') let userId = null