Timestamping Files or Strings

We can get a free API key and then, thanks to OriginStamp’s API, write a simple Bash shell script using curl & sha256sum to timestamp files or strings, which we will name timestamp, make executable with chmod +x timestamp, and put somewhere in our path:

#!/bin/bash
set -euo pipefail

API_KEY="73be2f5ae81ffa076480ac4d48fa9b2d"

# loop over input targets, hash them whether file or string, and submit:
for TARGET in "$@"; do

 if [ -f "$TARGET" ]; then
  # since it's a valid file, tell `sha256sum` to read it and hash it:
  HASH=$(sha256sum "$TARGET" | cut --delimiter=' ' --fields=1)
 else
  # if it's a string we're precommitting to instead, pipe it into `sha256sum`:
  HASH=$(echo "$TARGET" | sha256sum | cut --delimiter=' ' --fields=1)
 fi

 echo -n "$TARGET: "
 curl --request POST --header "Content-Type: application/json" --header "Authorization: Token token=$API_KEY" \
      --data "{\"hash_sha256\":\"$HASH\"}" 'https://originstamp.com/api/stamps'
 # print newline to keep output tidier since curl doesn't add a final newline to the JSON output
 echo
done

Now we can timestamp arbitrary files or strings as we please:

$ timestamp ~/wiki/catnip.md ~/wiki/image/logo/logo.png
# /home/gwern/wiki/catnip.md: {"hash_sha256":"4b357388100f3cdf330bfa30572e7b3779564295a8f5e6e695fa8b2304fa450e",
#  "created_at":"2015-12-02T23:57:56.985Z","updated_at":"2015-12-02T23:57:56.985Z","submitted_at":null,"title":null}
#
# /home/gwern/wiki/image/logo/logo.png: {"hash_sha256":"243d5b9b4f97931a07d02497b8fddb181f9ba72dc37bd914077e3714d0163a2f",
# "created_at":"2015-12-02T23:57:20.996Z","updated_at":"2015-12-02T23:57:20.996Z","submitted_at":null,"title":null}

$ timestamp "Lyndon Johnson was really behind the Kennedy assassination." "Sorry: I ate the last cookie in the jar."
# Lyndon Johnson was really behind the Kennedy assassination.: {"hash_sha256":"4aef69aeaf777251d08b809ae1458c1b73653ee5f78699670d37849f6f92d116",
# "created_at":"2015-12-02T23:58:57.615Z","updated_at":"2015-12-02T23:58:57.615Z","submitted_at":null,"title":null}
#
# Sorry: I ate the last cookie in the jar.: {"hash_sha256":"508190d52a6dfff315c83d7014266737eeb70ab9b95e0cab253639de383a0b44",
# "created_at":"2015-12-02T23:59:03.475Z","updated_at":"2015-12-02T23:59:03.475Z","submitted_at":null,"title":null}

Timestamping Version Control Systems

Given this script, we can integrate timestamping elsewhere - for example, into a Git version control system repository of documents using its post-commit hook feature. We could write out the full curl call as part of a self-contained script, but we already factored the timestamping out as a separate shell script. So setting it up and enabling it is now as simple as:

echo 'timestamp $(git rev-parse HEAD)' >> .git/hooks/post-commit
chmod +x .git/hooks/post-commit

Now each commit we make, the SHA-1 hash of the last commit will be timestamped and we can take the repo later and prove that all of the content existed before a certain day; this might be source code, but also anything one might want to track changes to - interviews, web page archives, copies of emails, financial documents, etc.

This approach generalizes to most version control systems built on cryptographic hashes as IDs, where timestamping the ID-hashes is enough to assure the entire tree of content. (I’m not sure about other VCSes; perhaps the post-commit hooks could timestamp entire revisions/patches?)

There have long been concerns that SHA-1 is increasingly weak; as of 2017, collisions can be generated at feasible costs, so timestamps of SHA-1 hashes no longer prove as much as they used to.

It might be possible to use a tool like git-evtag for hashing the entire repository history including the changes themselves (rather than just the IDs), and timestamp this master hash instead of the latest-revision hash. Alternately, since there are no worries about SHA-256 being broken anytime soon, one could write a post-commit script to directly parse out a list of modified files & timestamp each file; in which case, every version of every file has its own separate SHA-256-based timestamp. (The disadvantage here is also an advantage as it enables selective disclosure: if you are timestamping the entire Git repository, then to subsequently prove the timestamp to a third party, you must provide the entire repository so they can replay it, see what the final state of the relevant file is, and check that it contains what you claim it contains and that the relevant revision’s SHA-1 is correctly timestamped; but if you have timestamped each file separately, you can provide just the relevant version of the relevant file from your repository, rather every version of every file prior. The tradeoff here is similar to that of timestamping a hash of a batch vs timestamping individual hashes.

Probably the best approach is to timestamp each file at the beginning, use VCS timestamps subsequently for regular activity, and every once in a long while timestamp all the files again; then for slow-changing files, one will be probably be able to reveal a useful timestamp without needing to reveal the whole VCS history as well, while still having backup timestamps of the whole VCS in case very fine-grained timestamps turn out to be necessary.)

Timestamping Downloaded Web Pages

Automatically tracking Git commits is easy because of the hook functionality, but what if we want to download web pages and then timestamp them? Downloading them normally with wget and then manually calling timestamp on whatever the file winds up being named is a pain, so we want to do it automatically. This gets a little trickier because if we write a script which takes a URL as an argument, we don’t necessarily know what the resulting filepath will be - the URL could redirect us to another version of that page with different arguments, another page on that domain, or to another domain entirely, and then there’s the URL-decoding to deal with.

The simple (and stupid) way is to parse out a filename from the wget output because it conveniently places the destination filename in a pair of Unicode quote marks, which give us a perfect way to parse out the first² downloaded filename; this turns out to work well enough in my preliminary testing of it. A script wget-archive which does this and works well with my archiver daemon:

#!/bin/bash
set -euo pipefail

cd ~/www/

USER_AGENT="Firefox 12.4"
FILE=$(nice -n 20 ionice -c3 wget --continue --unlink --page-requisites --timestamping -e robots=off \
                                  --reject .exe,.rar,.7z,.bin,.jar,.flv,.mp4,.avi,.webm,.ico \
                                  --user-agent="$USER_AGENT" "$@" 2>&1 \
       | grep -E 'Saving to: ‘.*’' | sed -e 's/Saving to: ‘//' | tr -d '’' | head -1 )

timestamp "$FILE"

“Difficulties of Timestamping Archived Web Pages”, Aturban et al 2017, discuss some of the limitations of timestamping systems and points out a problem with this shell script: only the downloaded file (usually the HTML file) is timestamped, and the files necessary for properly displaying it (like JS or images) are downloaded but not timestamped; these additional files could change (either for normal reasons or maliciously) and change the appearance or content of the main HTML file. They provide a modified shell script (sha256_include_all.sh) to include the auxiliary files as well:

#!/bin/bash
rm -rf ~/tmp_www/ # [use of `mktemp` would be better]
cd  ~/tmp_www/

USER_AGENT="Firefox 6.4"

FILE=$(nice -n 20 wget --continue --unlink --page-requisites
                       --timestamping -e robots=off -k
                       --user-agent="$USER_AGENT" "$1" 2>&1
    | grep -E 'Saving to: .*'
    | sed -e 's/Saving to: //' | tr -d '’')

let "c=0"
for TARGET in $FILE; do
    if [ -f "$TARGET" ]; then
        let "c++"
        CONT=$(cat $TARGET)
        HASH=$(echo "$CONT" | shasum -a 256 | awk 'print $1;')
        echo "$HASH" >> "allhashes.txt"
    fi
    done

if [ $c = 1 ]; then
    FINAL_HASH="$HASH"
    else
    FINAL_HASH=$(cat "allhashes.txt" | shasum -a 256
                 | awk 'print $1;')
fi
echo "Final hash: $FINAL_HASH"

Example usage:

$ sha256_include_all.sh \
http://wsdl-maturban.cs.odu.edu:11011/michael/wayback/20170717185130/https://climate.nasa.gov/vital-signs/carbon-dioxide/
# Final hash: 2fa7ece06402cc9d89b9cfe7a53e4ec31a4417a34d79fee584c 01d706036e8cb