After trying to deal with relative imports and reading the many StackOverflow posts about it, I'm starting to realize this is way more complicated than it needs to be.
No one else is using this code but me. I'm not creating a tool or an api, so I see no need to create a "package" or a "module". I just want to organize my code into folders, as opposed to having a 50 scripts all in the same directory. It's just ridiculous.
Basically, just want to have three folders and be able to import scripts from wherever I want. One folder that contains some scripts with utility functions. Another folder has the majority of the code, and another folder that contains some experiments (I'm doing machine learning research).
/project/code/ /project/utils/ /project/experiments/
All I need to do is import python files between folders.
One solution I have found is putting
__init__.py files in each directory including the root directory where the folders live. But I then need to run my experiments in the parent of that directory.
/experiment1.py /experiment2.py /project/__init__.py /project/code/__init__.py /project/utils/__init__.py
So the above works. But then I have two problems. My experiments don't live in the same folder as the code. This is just annoying, but I guess I can live with it. But the bigger problem is that I can't put my experiments different folders:
/experiments/experiment1/something.py /experiments/experiment2/something_else.py /project/__init__.py /project/code/__init__.py /project/utils/__init__.py
I suppose I could symlink the project directory into each experiment folder, but thats ridiculous.
The other way of doing it is treating everything like a module:
/project/__init__.py /project/code/__init__.py /project/utils/__init__.py /project/experiments/__init__.py /project/experiments/experiment1/something.py /project/experiments/experiment2/something_else.py
But then I have to run my experiments with
python -m project.experiments.experiment1.something which just seems odd to me.
A solution I have found thus far is:
import imp import os currentDir = os.path.dirname(__file__) filename = os.path.join(currentDir, '../../utils/helpful.py') helpful = imp.load_source('helpful',filename)
This works, but it is tedious and ugly though. I tried creating a script to handle this for me, but then the
os.path.dirname(__file__) is wrong.
Surely someone has been in my position trying to organize their python scripts in folders rather than having them all flat in a directory.
Is there a good, simple solution to this problem, or will I have to resort to one of the above?Answer1:
Running python files as a module seems also odd to me. You can put your experiments in a folder still by putting a
main.py file on the root directory. So the folder tree would be the following;
/project/experiments /project/code /project/utils /project/main.py
Call your experiments on your
main.py file and make your imports on
main.py file. The
__init__.pys should also be inside each folder.
By this way, you won't need to run the py files as a python module. Also, the project will have a single entry point which is very useful in many cases.Answer2:
So I found <a href="http://www.scotttorborg.com/python-packaging/minimal.html" rel="nofollow">this</a> tutorial very useful.
The solution I have come up with is as follows:
project/ .gitignore setup.py README.rst MANIFEST.in code/ __init__.py something.py tests/ __init__.py tests.py utils/ __init__.py utils.py experiments/ experiment1/ data.json experiment1.py experiment2/ data.json experiment2.py
Then I run
python setup.py develop in order symlink my code so I can import it anywhere else (you can unlink with
python setup.py develop --uninstall).
I still haven't decided whether I like my experiments to live inside the project folder or outside. I don't think it really matters since this code is only for my personal use. But I suppose it would be proper for it to live outside...