Ttdfjt

I started my PhD 7 months ago, and as I generate more and more data, and do more and more analysis, my folder structure is getting out of hand. I wanted to ask for best practices and opinions on how to organize my files in order to not lose track of everything and quickly find what I need. I am a geoscientist and have lots of analytical data as well as programming scripts. Along that I also have README.md for some analysis and snippets of ms word or plain text, if I wwrite something down to remember for a paper possibly.

I've made it a habit not to edit raw data at all and to backup regularly for obvious reasons.

Right now my simplified general structure looks a bit like this:

├── data
│   ├── analysis
│   │   ├── isotope_temperature_reconstruction
│   │   │   ├── report.ipynb
│   │   │   └── script523.py
│   │   └── light_micro_growth-line-analysis
│   │   ├── img1.svg
│   │   └── regression.py
│   └── raw
│   ├── isotopes
│   │   ├── run2019_02_19
│   │   └── run2019_02_24
│   ├── light_microscope
│   │   ├── sample_xy123123
│   │   └── sample_xy123124
│   └── sem
│   ├── sample_xy123123
│   └── sample_xy123124
└── documents
 └── paper1

I know the system my files are organized in doesn't really matter as long as it is consistent. However I am facing some struggles:

• The data usually varies in "quality" and "ripeness". I have data that resulted from:
  
  
  • Some trivial test -> won't be used ever again
  
  
  • Is for calibration -> Doesn't belong to any project, but matters in many cases
  
  
  • Is directly and only needed for a certain publication
  
  

My problems with this are:

• I find myself often linking and copypasting my data all over the place, because it is not where it's currently needed. As a consequence i also make edits only in certain places and not everywhere, and lose track of whats the most recent file. I also lose track of what files are handled programatically and where I edited something "per hand".


• I have a lot of duplicate python/R/whatever scripts that I copypaste wherever needed. I think this is the easier part to resolve by modulating code and putting it into version controlled system wide libraries.


• I sometimes have snippets of word or plain text that contain relevant research insights, but are scattered all over the place, because they are not directly related to a paper / data.

So I am looking for suggestions to address these problems, as well as suggestions for general file and data management, and general organization at a researchers main desktop machine.

The only problem that I feel adequately solved is my literature management because I just use zotero and let it organuize all my papers in a coherent folder structure. (It also makes it easy to search via tags, which would be super cool for data files)

In general I would distinguish between two approaches or a combination of both before ending in a very time consuming categorization of all your files

• 
  

  top-down

  
  
  

  This means you use intelligent software to find again files you are looking for and don't worry too much about where it is stored. For example reference manager (Mendeley, Zotero,...), a Desktop search engine (Copernic). In the desktop search engine or with windows search (indexing turned on) you should see which file was changed at the latest. Most of your ideas and sketches you save in Onenote or something else that can link to files and www links.

  
  


• 
  

  bottom-up

  
  
  

  This means you come up your self with a distinct structure like you did with a rough categorization (raw data, devices, projects). But don't complexify this, when most of the files can be identified by the file type/ending, then it may be more convenient to order it by projects and keep all file types within such subfolders.

  
  

Additionally, one trick I use is tagging files in their filename like "#phdthesis" or "#interesting" or "#cite" or "#collaboration".

Sample files (data, images of samples) always get a date ProjectnameDayMonthYear independent of their file type.

Don't make your system too complex or too time consuming. For instance most of PDFs I read and download end in one folder. No point in categorizing them, this will not work/help over years and decades. Defining content-relative filenames takes too much time, most of them I find again over my desktop search engine months or years later if they were memorizable with keywords and search operators or self-created #tags.

For files which the PI or teammembers have to have the possibility to retrace them you anyway have to arrange a common plan how to structure and save the files everyone understands and can contribute. I would either suggest cloud storage or a local wiki here (moinmoinwiki for instance).

The most productive system is probably the one "invented" by Stephen Wolfram, but it took probably even more time to set it up than writing his article. But you get a lot of ideas, so I'm not sure it is future-proof to organize yourself with a propertiary software.

A personal opensource free wiki like moinmoinwiki or tiddlywiki is also an alternative and combination of bottom-up top-down and file system. So my experience is a wiki is only worth the additional time and effort at least with 2 or more team members, otherwise your personal choice and mixture of bottom-up and top-down is more time-efficient and humanities/experimental research/programming single reaserchers will end up with very different management system due to available time, number of files, work-flow and necessity to document and retrace everything.

File systems and desktop search engines will never die out, so my general tip is not to rely too much on fancy software like onenote, mathematica or alternatives to have a future-proof system.

One of the more important goals when organizing your scientific data is to make it reproducible. At some point someone will have to look at the data and analysis again and try to understand it. This will often be you, e.g. when writing papers or your thesis, but also other scientists later that continue to work and expand on your projects. If you create a figure and your PI or a reviewer asks how exactly it was created and which datasets its based on, you should be able to figure that out from your data.

This requirements means that copy and pasting is not necessarily a bad thing. You want to be able to reproduce your analysis with exactly the flawed scripts you used the first time, not the updated version that might fix bugs there or might have introduced new ones. But you also want to be able to update your analyses if you find a problem in one of your common scripts, and check if your results changed due to this.

If your data isn't very large, I'd simply duplicate it in every analysis folder. That way you have both the code and the data in the same place when you have to revisit it. If it is large, you should take care to create an immutable raw data storage that you link to, and make sure those links are never broken by renaming or removing data.

For your scripts, it seems like you are experiencing issues because you haven't centralized your common code. Creating a modular version of your common code like you suggested is certainly a good idea. But I would run this in a way that copies your common code as it is at that time to the current analysis folder. So that you can just rerun the analysis with the old code easily, but still have a way to also run it with updated common code, if desired.

If you have problems distinguishing where you have edited stuff by hand, or where you copied stuff, it seems like you're doing this in multiple directions. It's always easier to follow this if your changes only flow in one direction. It's also a good idea to put as much of the data and code as reasonable into a real version control system like git. So you can just use e.g. "git blame" to figure out if you edited some parts of a script manually.

For finding snippets of text, I'd probably just use some kind of full text search. Maybe use a naming convention for this kind of files, so that you can distinguish them from your data and code files, if you want to search in them.