Welcome to Hashdex’s documentation!¶

Contents:

Hashdex¶

A file indexer based on content hashes to quickly find duplicate files on your system. I created this tool because I always forget which pictures from my phone I already uploaded to my Dropbox account. Now I can just upload all pictures to an uploads directory check these files against the index and only organize the remaining files.

As easy as..¶

pip install hashdex
hashdex add /path/to/my-main-pictures-directory
hashdex check --rm /path/to/my-uploads-directory

Free software: MIT license
Documentation: https://hashdex.readthedocs.io.

Features¶

create an index of your files
find duplicate files on your filesystem
check if files in a directory are already indexed

Credits¶

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Installation¶

Stable release¶

To install Hashdex, run this command in your terminal:

$ pip install hashdex

This is the preferred method to install Hashdex, as it will always install the most recent stable release.

If you don’t have pip installed, this Python installation guide can guide you through the process.

From sources¶

The sources for Hashdex can be downloaded from the Github repo.

You can either clone the public repository:

$ git clone git://github.com/jeff-99/hashdex

Or download the tarball:

$ curl  -OL https://github.com/jeff-99/hashdex/tarball/master

Once you have a copy of the source, you can install it with:

$ python setup.py install

Usage¶

To use hashdex with docker run the image like

docker run \
    --rm \                                 # remove container after usage
    -v "/path/to/root/directory:/media" \  # mount the root directory of the files to index or check
    -v "/path/to/index:/db" \              # mount the output directory or file for the index database
    jeff-99/hashdex:latest \               # docker image name
    add \                                  # command
    --index /db/name-of-index.db \         # index location should always start with '/db'
    /media                                 # location of files should always start with '/media' mounted dir

Hashdex is a command-line utility for quickly checking your files for duplicates

Create an Hashdex¶

To initialize an hashdex you need to add some files/directories to the index..

# directory
hashdex add /path/to/directory

# file
hashdex add /path/to/file.txt

This will create an index file in the users home folder (~/.config/hashdex/index.db) If you want to create an index in another location you can specify the –index option to the command

hashdex add --index /path/to/index.db /path/to/directory

Once you have added all the necessary directories to the index we can begin checking directories against the index. To check all files in a directory against the index execute the following

Check new files against index¶

# directory
hashdex check --index /path/to/index.db /path/to/directory/to/check

# file
hashdex check /path/to/file.txt

This will list all files in the given directory which are already indexed with the indexed file path. You can add the –rm flag to delete all files in the given directory which are found in the index, so you will be left with only new files. In addition to the –rm flag you can also pass an –mv option with an existing path to move duplicate files to the given directory.

# remove duplicates
hashdex check --rm /path/to/directory/to/check

# move duplicates
hashdex check --mv ./duplicates /path/to/directory/to/check

Find duplicate files¶

To get a list of all duplicate files within already indexed files you can execute:

hashdex duplicates --index /path/to/index.db

Cleanup the index¶

Overtime files will be deleted from your system or moved to another location. So once in a while you should clean up the index with the following command:

hashdex cleanup --index /path/to/index.db

Contributing¶

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

You can contribute in many ways:

Types of Contributions¶

Report Bugs¶

Report bugs at https://github.com/jeff-99/hashdex/issues.

If you are reporting a bug, please include:

Your operating system name and version.
Any details about your local setup that might be helpful in troubleshooting.
Detailed steps to reproduce the bug.

Fix Bugs¶

Look through the GitHub issues for bugs. Anything tagged with “bug” and “help wanted” is open to whoever wants to implement it.

Implement Features¶

Look through the GitHub issues for features. Anything tagged with “enhancement” and “help wanted” is open to whoever wants to implement it.

Write Documentation¶

Hashdex could always use more documentation, whether as part of the official Hashdex docs, in docstrings, or even on the web in blog posts, articles, and such.

Submit Feedback¶

The best way to send feedback is to file an issue at https://github.com/jeff-99/hashdex/issues.

If you are proposing a feature:

Explain in detail how it would work.
Keep the scope as narrow as possible, to make it easier to implement.
Remember that this is a volunteer-driven project, and that contributions are welcome :)

Get Started!¶

Ready to contribute? Here’s how to set up hashdex for local development.

Fork the hashdex repo on GitHub.

Clone your fork locally:

$ git clone git@github.com:your_name_here/hashdex.git

Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:
```
$ mkvirtualenv hashdex
$ cd hashdex/
$ python setup.py develop
```
Create a branch for local development:
```
$ git checkout -b name-of-your-bugfix-or-feature
```
Now you can make your changes locally.
When you’re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:
```
$ flake8 hashdex tests
$ python setup.py test or py.test
$ tox
```
To get flake8 and tox, just pip install them into your virtualenv.

Commit your changes and push your branch to GitHub:

$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature

Submit a pull request through the GitHub website.

Pull Request Guidelines¶

Before you submit a pull request, check that it meets these guidelines:

The pull request should include tests.
If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst.
The pull request should work for Python 2.6, 2.7, 3.3, 3.4 and 3.5, and for PyPy. Check https://travis-ci.org/jeff-99/hashdex/pull_requests and make sure that the tests pass for all supported Python versions.

Tips¶

To run a subset of tests:

$ py.test tests.test_hashdex

Credits¶

Development Lead¶

Jeffrey Slort <j_slort@hotmail.com>

Contributors¶

None yet. Why not be the first?

History¶

0.6.1 (2020-11-18)¶

Updated development dependencies
Added python 3.8 support
Dropped python 2.7, 3.4 support
Added Dockerfile

0.6.0 (2017-10-28)¶

Added version option to main command
Added option to move files while checking for duplicates
Updated pytest-runner to 3.0.0
Updated cryptography to 2.1.2
Updated sphinx to 1.6.5
Updated flake8 to 3.5.0
Updated tox to 2.9.1
Updated pytest to 3.2.3

0.5.0 (2017-09-28)¶

added support for adding and checking files instead of directories
moved default index file to users home directory
Updated dependencies

0.4.1 (2017-09-24)¶

adjusted for python 2.7 math.floor returning a float

0.4.0 (2017-09-24)¶

renamed the index command to ‘add’
refactored –dir option for add/check command to cli argument
added cleanup command
added docs
updated dependencies

0.3.0 (2017-09-12)¶

Added –rm flag to check cli command (without the flag the command will not delete files)
Dropped python 2.6 & 3.3 support
Updated dependencies

0.2.0 (2017-09-08)¶

Refactored Indexer class
Added Hasher class
Added tests 100% code coverage
Updated dependencies

0.1.1 (2017-09-05)¶

Updated dependencies
Fixed file listing bug

0.1.0 (2017-09-04)¶

First release on PyPI.

Welcome to Hashdex’s documentation!¶

Hashdex¶

As easy as..¶

Features¶

Credits¶

Installation¶

Stable release¶

From sources¶

Usage¶

Create an Hashdex¶

Check new files against index¶

Find duplicate files¶

Cleanup the index¶

Contributing¶

Types of Contributions¶

Report Bugs¶

Fix Bugs¶

Implement Features¶

Write Documentation¶

Submit Feedback¶

Get Started!¶

Pull Request Guidelines¶

Tips¶

Credits¶

Development Lead¶

Contributors¶

History¶

0.6.1 (2020-11-18)¶

0.6.0 (2017-10-28)¶

0.5.0 (2017-09-28)¶

0.4.1 (2017-09-24)¶

0.4.0 (2017-09-24)¶

0.3.0 (2017-09-12)¶

0.2.0 (2017-09-08)¶

0.1.1 (2017-09-05)¶

0.1.0 (2017-09-04)¶

Indices and tables¶