Ask HN: How do you store the knowledge gained in a day?
Each day I (and I assume most knowledge workers, devs, creatives) read many articles, papers, code snippets, AI responses, discord messages etc.
At the end of the day some of this information is most likely lodged in your brain and the digital version can be discarded. However some of it should be retained manually in some system - or at least I feel it should.
What approaches do people use to consolidate and store this information to allow all tabs etc to be closed for the next work day?
I'm also liking obsidian a lot, especially that the data lives on the filesystem and in markdown so it's not walled in by obsidian. I made my obsidian project folder a git repository and have a script that runs a few times a day to check for changes, if there are any, it creates a backup commit, and pushes the changes to a remote machine. What initially drew me to obsidian was the graph showing connection between documents, and while most are connected, it's not what made me stay.. What made me stay was the familiar-ish interface (reminds me of vscode, which reminds me of "ultra edit") and the way it does not impose too much structure on me.
You don't, because information hoarding is only slightly less bad than real-life hoarding.
Be honest with yourself: other than the occasional search, when is the last time you referred to notes you took X years ago? And for the search-cases, how outdated was that information? Is the sequence of commands you painstakingly saved a decade ago for how to rebuild an array in megaraid really relevant anymore? Could you not have just, ya know, just googled it next time you needed it? Is your written-out explanation for how heat pumps work from that time you were re-doing your home heating system really useful.. when you can just have a conversation with Gemini about it now?
Less is more.
I don't delete files. They are put in directories, and at some interval, the directories are put in other directories. I've kept everything that didn't die in sudden catastrophic failures. I can access files that I got before there was even a harddrive in my computer. And I love it, like browsing an old photo album, I once in a while take a stroll through my life, but, unlike the photo, which only captures one angle of a situation, I have entire installations to look at, what was on my desktop 20 years ago, what had I downloaded, oh, that movie, yeah, I remember it now! those funny pictures a long-forgotten friend send to me over IRC DCC, the chatlogs of our cringe teenage banter..
Yes, this is hoarding, but, as opposed to the physical kind, where it goes from some neatly displayed trinkets, to inaccessible rooms and biological hazards, computer files allow us almost infinite room, and almost infinite structure.
While it is probably not healthy to entertain the idea, I believe most hoarders, given sufficient physical space (infinite, really, is sufficient cannot be determined before they're dead and no longer amassing objects), their collections, their order, would look somewhat neater (although not any less pathological).
Whereas physical objects are a burden both in weight and volume, a few hundred terabytes of data does not take up significantly more room in the physical realm.. A file server, and its backup, is really all it takes..
So I don't think they're comparible.. At least, it depends, also, how it affects you, is it a source of stress and negativity, or simply a peace of mind, or a mad obsession..
I don't delete stuff, but I don't fuss over it either, it's not a big deal, once in a while, I move everything out of sight, it's a very small effort, and, so far, 30 years later, current me is grateful for past me for leaving those little breadcrumbs to spark my memories, and so, I will continue this tradition, in honor of future me.
Working in management, OneNote is my second brain I couldn't do without. Pages of meeting notes everyday, documentation of internal processes etc., all of which I regularly look up.
Obsidian has been really positive for me. It has paid dividends for years — I look up notes I made years ago pretty regularly. I highly recommend it. I used Anki on and of with benefits but without enough discipline. Good thing with Obsidian is that you can write and forget about it - your obscure notes about http2 multiplexing will still be there.
One thing to remember: most stuff can be forgotten safely. There are very few things you’ll actually want to look up a few months from now on…
Yes, but the process of creating the notes, storing ideas and connections you made during a day is useful right?
well, if you're using the process of creating notes as a means for learning, handwritten notes would probably be more beneficial
that's a myth; while it's true that handwriting does lend to better recall, it's the process of breaking down what to write down (thus spending more time making sense of it) that helps you remember.. not the 'muscle memory' everyone talks about. You can do the same thing in digital tools (typing out notes, structuring notes, etc) and sometimes you can do it faster.
I use a combination of Obsidian notes and Confluence.
Obsidian is a not-very-organized mess unfortunately; I still haven't figured this out but it's good enough for me to usually find things I remember. Confluence is organized as such https://docs.divio.com/documentation-system/. This works great.I always know where to put something and where to find it again later.
I also don't keep tabs open, I write things on a Todo list in Obsidian. Then at the beginning and end of the day I organize it; finishing small things like "record this", prioritizing, and moving project specific tasks to a page just for that project.
Honest suggestion: do not worry about where to put it, worry about what you will type into the searchbar when you half-remember it 2 years later.
No file structure is ever gonna beat a good searchbar, adjust your own notes accordingly. I completely hid the file explorer. If I need to cluster things together, I use very specific key words at the top of the file (think: #Parent/Child), they're far more flexible than a file system.
I have been exploring using tags in file properties instead of a folder layout. Just so I can pull things like they are a person, they worked for this company. Or I'm only looking for project docs, or article summaries. I just haven't wholesale moved so I'm getting the disadvantages of both approaches :p
I generally think you're right though so I am moving away from folders. Especially when the notes are raw text (markdown) and can be searched by anything effectively.
I am also interested in graph searching but I haven't used it yet.
I’ve been using obsidian for this for the past couple of years. I have lots of folders and notes for work projects along with notes for learning topics. I keep track of tasks and to do lists for me and people in my team (using #people/name tags).
I have specific updates notes for each project. I add a weekly header (via a template) in reverse chronological order. Pull the key updates into that (I also have a todo for each project that pulls in tasks from all files in that project subfolder - so picks up meeting actions)
I also put them all in a canvas so when I doing updates about projects they’re all in one place.
I do the same for people I manage so I can keep track of updates and todos when I’m meeting with them.
My go to start point is a daily not where I throw in new tasks (tagged to projects or people) and those get pulled into the corresponding project/person file.
Because it’s markdown with lots of plugins it helps me do things like diagrams (mermaid diagrams) etc and excalidraw/canvas for architecting/mind map style ideation.
I use a flat text file called "notes" on my desktop, and I leave it open in Sublime Text in a corner of my second screen. Periodically I throw a datestamp in there as a reference point. For generic "stuff that should be retained manually" it works well - easy to add to, easily searchable.
Could you provide a small example of what you may add in a day and how you later search it?
Half of the battle is avoiding over-documentation - I only write things here that I know I'll want to remember later: for me these are important notes, figures I hear/see in meetings, critical names of people, that kind of stuff.
Mostly it's just bullets. If a bullet is critical I'll prepend it with !!!. If it's a task, I'll use * instead of - for the bullet marker (then move the task to my separate task tracker later - notes doc isn't good for long-term task management)
If I want to tag the note explicitly with a "tag", I'll add one in brackets - but honestly these aren't that helpful.
Ctrl-F will generally find what I need. Aside from that, what's mostly been helpful is the dates - I often remember roughly when I learned a bit of information, so if I can't find it with Ctrl-F, I'll scroll back to that "era" and look around.
Pay a lot of attention to yourself when you're looking for historical information: what do you search for? Are there key things you often look up over and over? If so, what search terms did you use when trying to find them? Add those explicitly to the entry, so next time you'll find it easier.
Any thing at all.
Ctrl-F.
This is it...
KIS
Ctrl-Effing-F
Any other trendy nonsense is time-wasting trifling foolishness.
I just note the solution for something if it’s likely I need it again and the solution took a while to find.
I don't accumulate tabs (I rarely have more than two open at a time), so I don't have to do anything to allow them to be closed. I just close the browser.
With your larger question, though, if there's specific information that I feel the need to keep locally, I add that to a wiki that I run. If information is in a non-online form (books, etc.), all I need to do is remember that it's in there. I don't need to remember the information itself because I can look it up at will. For everything else, I don't worry about it at all. If I found it online once, I can find it online again should I need it again.
Ohh, I really wish I also had this attitude (YAGNI)!
But do you not believe there is some value in trying to store some of what you found / learnt each day, to allow you to make some new connections in the brain the following days?
> do you not believe there is some value in trying to store some of what you found / learnt each day,
Sure, but I retain some of that without having to do anything special. I have no need to remember everything. My brain seems to know better than I do what is worth remembering and what isn't anyway. I trust it.
Retention is even more likely to happen if it's information that I actually used that day, and if it's really valuable information, then I probably have used it immediately.
Org mode versioned with git.
Every issue at work becomes an item in this org, and all information relevant to the issue goes there: emails, chats, code snippets, documentation paragraphs, etc...
It's amazing how clearer things become, and how quickly you can get back in action if the issue resurfaces a few years later.
Currently I use Obsidian to make note of a few key things, but what I'm more excited about is that the way tech is evolving is going to allow us much more expansive solutions for personal knowledge-bases. Eg: See what Perkeep tried to do.
Here's an idea: We should be able to record the full experience stream (or just the laptop screen, let's say) for a person -- digitize it suitably (eg. pull out all the text, or tokenize it with a VLM, etc), attach some metadata, and have the whole database ready for a person to query / play with, using powerful LLMs.
To the extent that you can store everything and perform effective recall/search, it obviates the need to carefully "pre-process" bits by noting them down in the right file, tagging them appropriately, adding metadata, etc. All of which should make for a much nicer UX.
Given the Jevons' paradox though -- I wouldn't be surprised if our experience stream becomes so dense/rich that storing everything and querying from it eventually becomes prohibitively expensive. We would then have to construct pre-processing rules to prune the stream, and amortize the cost of certain deductions by performing them at data ingestion time instead of repeating it for each query. It's all just the basic principles of system design at the end of the day, and how systems need to be rearranged as various component capabilities (and user needs) scale.
I get why this may seem attractive, but my view is that we need the manual step there as a way to force our brains to remember things.
What you suggest is a nice UC for sure, but not sure it will make any of us smarter in the long run.
I have a discord bot and a bunch of channels for projects and a new thread is started weekly for that weeks notes. The bot reads and responds to messages posted there and saves them to the appropriate location and organizes them for me
This is why I write stuff in reMarkable.
At some point, I have to type it in so that I can share with others. This transfers short term memory into long term.
I know there are a billion notes systems that have heaps of features. Nothing beats writing for me.
I wish I had something like this earlier in my career. I've forgotten so much useful info that only comes back when the right external triggers are fired. It's annoying.
I use browser bookmarks organized into folders like /dev/blog /dev/tools etc. so that I can browse by category later when I think “what was that tool again?” A flat list of bookmarks drives me insane, all I want to do is delete it.
I tried Obsidian for a while but I’m too prone to bike shedding with it, so I gave up. I DM myself in Slack with bits of info to keep around for the near term, because it’s already a simple feed ordered by time.
I know the feeling of the bike shedding with the tools!
This is cool for links, but often I also want some nuggets/ideas to be saved.
I use Obsidian for kanban (with urgent tasks) and for drafting longer writing, while using Remnote for tracking most other general information. Both work really well for me, for different purposes. I use Obsidian more as a visual information processing medium.
I will say, it's really helpful to open up excalidraw through Obsidian and create a quick flow chart or drawing with my team.
If it's interesting enough, and/or "big, will read later" then I save a local copy (pdf, html, plain text or even an image).
Mostly organised in logical-to-me directory structure.
Local storage beats online-only in many ways, imho. Online content rots away over time. Whereas backing up local files is easy.
Beside that: browser bookmarks, simple note-taking app, and (last but not least) pen & paper.
Consolidate? All in a single place? Information comes from a lot of sources, in a lot of ways to be accessed, and a lot of ways to be shared and with who.
A lot are notes or links collections in obsidian. Others may be canvas on slack channels or conversations, some cloud note taking apps depending on with who I am sharing or collaborating on them, bookmarks in browser or RSS reader, even chats with ChatGTP.
In general it depends on where I got that piece of information or knowledge, what I plan to do with it, who I want to share it with and so on. There is no central consolidation process for all the diverse kinds of information, as there are "right" places to have them depending on it.
> However some of it should be retained manually in some system - or at least I feel it should.
I question this premise. If the content you read on any given day did not the result in some change to your work product, it is really important enough to save it? Do you really need to remember all the messages, chats, articles that didn't give you any actionable answers? Or do you just keep the results that you created based on such things?
I keep none of it. I do, however, keep all my old code. Every throwaway line I've ever written is saved somewhere, even if it is just in git history on old throwaway projects. So when I do need to do something again, I don't need to re-do research or figure it out from scratch... I have my working solutions already stored.
I tried several "information management" and "knowlege management" programs over the years, but none of them worked the way I wanted.
My original method was a directory full of plain text files arranged by subject, that I brought up with my text editor.
Over the years I wrote a specification for a system tailored for my needs, that could manage different forms of media, notes, saved web pages, ebooks, and so forth. The spec grew until I set the project aside as more trouble than it was worth. In retrospect I wish I had kept going.
At the moment I'm using plain old hand-written subset of HTML linked to subdirectories arranged by subject. As time permits I will finish the (simplistic) custom browser, which will also let me annotate pages and edit files.
I have looked into some of the indexing programs that look inside files and index their contents in a searchable fashion. They all take a vast amount of storage space, probably because over the decades I have amassed a vast amount of data. After experimentation I decided to pass on the indexers as well. I have always managed things in a rigid hierachical arrangement, and it's pretty easy to find what I'm looking for. On the other hand, I know what's there and where I put it, which makes it easy for me. It would be much less useful for someone else.
Never underestimate the usefulness of a plain text file. It can be written, read, searched, and indexed by almost anything, should you wish to do so. Some of mine date back to the mid-1980s and MS-DOS 2.1, to OS/2 and Windows (briefly), Linux, and even Haiku. Every platform has subdirectories and some kind of editor; that's real portability. And if you ever have to deal with ancient machines that don't speak ASCII, much less UTF, there's probably some kind of conversion utility to let you move your files to IBM EBCDIC or DEC Sixbit.
If you're running emacs, the "org mode" add-in uses lightly-formatted text files organize data, which is then available within the editor. It's worth looking at if you're an emacs user, assuming you're not using org mode already.
For things online, I just bookmark web pages in a folder called "Other". For written materials/books, I write notes in the margins or write free-hand notes on a piece of paper and insert it in the book somewhere. Then when I'm bored, instead of infinite-scrolling some stupid app, I try to pick out of the "Other" folder or read a book off the shelf. At least, that's what I aspire to do most of the time...
I also decided that if something's worth remembering, I'll remember it. If it's not worth remembering, then I'm also not gonna both re-reading notes about it, no matter what note-taking tool or method I use...
Zettlekasten with git and https://github.com/foambubble/foam - if I'm mobile I use WorkingCopy to paste links into an inbox file that I organize later.
everyday i create a file if ~/self directory with name %dd-%mm-%Y.txt i put there everything, literally: code snippets, thought, todos, contacts, stack traces, links. i am in emacs, but i do not use org and git - have no idea why i should. searching is just a grep
I primarily use Logseq because I've found that outliners best align with my mental model. I mainly take notes within my daily journal. When I encounter something worth saving, I simply create a new entry. A typical day looks like this: https://i.imgur.com/wAiYgSY.png
To categorize the information, I simply link the entry to a specific page. For example, if I discover an interesting React library, I create a new entry like this: https://i.imgur.com/dDlStkZ.png
A great feature is that Logseq understands page hierarchies, so if I click on [[Dev/JS]], it presents a clear hierarchical overview: https://i.imgur.com/z9hGmmh.png
Using this approach, I've stopped bookmarking sites in my browser altogether. In Logseq, I can connect useful things to each other, add entire notes, and more, as shown here: https://i.imgur.com/Krld2cS.png
If I want to remember something, I just add a #card tag to the block, and it automatically syncs with Anki via a plugin.
Supermemo 18 and the 20 rules of knowledge formulation.
I will look it up for sure! It would be cool to elaborate a little directly here if you could, or at least a few links please.
Maybe this: https://www.supermemo.com/en/blog/twenty-rules-of-formulatin...
A git repo with text files (markdown) with names as tags separated by underscores (foo_bar_baz.md). Edit those with emacs. Periodically commit to be able to browse it by date.
Search by file tags:
Search by contents:I have one massive text file (well, actually an Apple Note) where I paste anything I need to reference throughout the day. Reverse chronological, newest stuff on top. If I need something later, it'll be in there. If the information is sufficiently important, it'll eventually move into another more focused document.
Is this just links or also ideas, connections etc?
All of the above. I throw everything in that one file to keep it simple and prevent myself from spending unnecessary time organizing information that I don't end up needing again in the future.
I suppose the exception is when I already have a project-specific doc set up, then links or thoughts related to that project will go in there. But even then, sometimes I use the default scratchpad first, and move the info later, the next time I'm doing dedicated work on that project.
Org mode.
I don't worry too much about the structure. So long as it's written down and backed up, it is searchable, and that's enough.
What sorts of things will you write? I am interested not just in the system but what you save.
Basically, anything and everything. Anecdotes. Who I spoke to and if they were nice or otherwise. Cool bits of technology I saw. How something works. What I did well, what I did wrong. I tend to include a date for context.
One file is just a time-ordered log where I put loose observations. Then for specific projects/themes I have separate files.
It is a bit of a mess, but it doesn't matter. It's a sort-of append only log.
I use the knowledge organizer I wrote at https://onemodel.org [AGPL], and write things wherever they fit in the deeply nested, searchable outline. Maybe it's like my own, efficient, highly-interlinked zettlekasten.
TIP: When asking for advice in relation to knowledge management, note-taking, etc., be sure to ask for precise details regarding commenters' solutions. There are many people who participate in these discussions who don't seem to take a lot of notes (e.g., one file or paper notepad for all of their notes!).
I have a personal knowledge base that currently includes almost 7,000 files in which I store my notes. I take notes on everything. Every technology. Every project. Every meeting. Every product I evaluate. EVERYTHING.
My notes are stored in Org files that I edit with Emacs and Org mode[1]. Org files are written using a feature-rich lightweight markup language[2] that is much more powerful than Markdown (which is used by other note-taking tools like Obsidian). For example, Org supports plain text spreadsheets[3], a feature I love.
People tend to disqualify Org and say, "I don't use Emacs," while assuming that Emacs users choose Org because we already use Emacs. But I started using Emacs specifically for Org, not programming.
Regardless of which tool you end up using, consider organizing your note files using hierarchical tagging. I started using hierarchical tagging for my notes right after Wikipedia launched and I saw how effectively hierarchical tagging was being used there. Each Wikipedia article can belong to multiple categories, and each category can belong to multiple categories. This is hierarchical tagging, and it's worked great for my notes. At the bottom of every one of my Org files, there is a list named "Parent topics", and each parent file has its own "Parent topics" list (excluding the "main topics" files, which have no parents).
[1] https://orgmode.org/
[2] https://orgmode.org/features.html
[3] https://orgmode.org/manual/The-Spreadsheet.html
I think it's impossible to retain the amount of information we consume.
I've personally given up and if I remember, that's great, if I don't, oh well. There is always more to learn tomorrow.
I created my personnal search engine to index and retrieve content I like. I like the idea of keeping track of content you like without too much effort.
https://github.com/raphaelsty/knowledge
I did not find yet a solution that suit me for knowledge which is not an online webpage / pdf.
Confluence for work. Personal space has a TIL and Personal notes on what I am working on. I leave them fully shared for Rovo and other staff to discover.
Every thing else is Confluence in team spaces. And Jira. Slack too.
Things can be harder to find in Slack so that is more a second resort after Confluence.
Downvoted! Should I have made something up?
Workflowy. This "list of lists" approach just works for me.
If it’s a link I find interesting, I take an excerpt of it and the link and post it on my personal blog. It’s not really even for public consumption as it is for both a public link blog for myself and where I post longer “thought pieces” that may later get copied to LinkedIn after I edit them a few times.
If I find an interesting link on HN, I’ll favorite it and post the link to the article and the HN discussion.
You have every right to throw up a little bit about the performative nature of LinkedIn. But in my niche, it’s the nature of the beast.
I do keep a copy of my career document where I note any major accomplishments and the details in STAR format for future projects, review times, 1x1’s and interviews in now a private Notion page.
DevonThink. Been using it for more than 10 years, tried other tools and none come close.
I have vscode open and a directory structure for organisation
The first folder - TODO is more like “Working memory” - unsorted interesting learnings, my actual todo list, notes and thoughts for the next few days
Then different sections for larger relevant parts of my life and memory
People Robots Company Code Reading ZanySchemes Family …etc
Whenever working memory is getting full I’ll take the time (30 seconds) to organise - either move or delete.
The key parts are - 1) Organise when it makes sense to, don’t try and design it all up front and don’t be scared of refactoring memories - the directory structure will anneal to something stable
2) have some damn discipline - tidy your notes, tidy your mind, and put the time and effort in, most people give up on note taking because they are too undisciplined to keep their virtual mind tidy, if you keep your notes tidy then up-keep is negligible, just like keeping your house tidy, maintain a high standard and then upkeep is easy
And these are just text files with no fixed structure?
Whatever it needs to be - some have pdfs, images, schematics, code, text files with links.
If a text file starts getting too big I’ll reorder it or split it out.
I think the key thing is developing the skill that suits you - don’t try and constrain it with some rule (all files must have structure X) because invariably something will not fit that pattern and then you’ll give up or spend too much effort trying to find the holy grail of data structures and then you’ll give up.
Let it flow naturally without constraints, it will settle on a form that is useful
I don’t bother.
You’ll almost certainly never look at 99.9% of it again so just move on.
I am currently leveraging scammers, doxxers and all sorts of scumbags to do that for me, for free. Better than AI.
I advised them to avoid it because it would be shameful for them to be used like a marionettes, but I guess if they stopped it would be even more shameful. There's something about their pride of being covert that makes it easy to use for this.
Sometimes it stores stuff that I don't care about, but that's fine.
I also do handwritten notes sometimes.
Sorry not sure I really understood your approach.
They use scam accounts and tactics to annoy me, I steadily shape their expectations to relay things that will work as reminders and all sorts of distributed storage.
Initially I didn't want to do it, but it's so reliable and predictable, that it became useful for that. I can't explain the exact mechanism either.
I still don't understand what you mean here. Do you have an example?
I don't have any example to share.
I also have problems storing some things. Like I wanted to store information about Tourettes and coprolalia, but it doesn't stick to my herd of scammers (maybe some of them have it and they're embarrassed?). That's the best example I can give (a non working example, but that's fine, no storage medium is perfect).