Pre-Receive Hooks
If you read my blog on Pre-Commit Hooks (if not, please read it), then you know about their limitations. To address them, GitHub Enterprise, and other VCS platforms, have an option to deploy pre-receive hooks and run them before accepting the push. This will equip organization and repository administrators with centralized control over what is being pushed to the repository.
Scenario
TachTech is developing an application that must meet specific standards, but most importantly, it must not contain any secrets (passwords, keys, PATs, etc.). To address that, we will deploy pre-receive hook on the organization and then activate it on the repository to meet the requirements.
Pre-Receive Hook Environment
Since we will be using a bash script to validate code pushes, we will not need a special pre-receive hook execution environment; the provided default will be more than enough, but for the completeness of this blog, I will show you how to create your own using two options; docker images/containers and chroot. Also, to justify the creation of such an environment, we will include the truffelhog binary in them as a tool to scan the commits for secrets.
Docker Image/Container Execution Environment
In this section, I will show you how to create a pre-receive hook execution environment using docker images. The only requirement for this part is to have docker installed on your machine. Now let’s take a look at our folder and the Dockerfile contents.
pre-receive-hook$ tree
.
└── Dockerfile
pre-receive-hook$ cat Dockerfile
FROM alpine:latest
RUN apk add --no-cache wget
WORKDIR /root
RUN wget https://github.com/trufflesecurity/trufflehog/releases/download/v3.34.0/trufflehog_3.34.0_linux_amd64.tar.gz && \
gunzip trufflehog_3.34.0_linux_amd64.tar.gz && \
tar -xvf trufflehog_3.34.0_linux_amd64.tar && \
mv /tmp/trufflehog/trufflehog /bin/
The Dockerfile will pull an alpine:latest image from the docker hub, install wget on it, then using wget will pull trufflehog tar file from GitHub, untar it and move it to the /bin/ folder so it can be called from the CLI.
Now let’s create the docker image, then a container, and then export it as a tar file. The whole process looks like this:
pre-receive-hook$ docker build -f Dockerfile -t pre-receive.trufflehog-3.34.0 .
<omitted for brevity>
pre-receive-hook$ docker create --name pre-receive.trufflehog-3.34.0 pre-receive.trufflehog-3.34.0 /bin/true
pre-receive-hook$ docker export pre-receive.trufflehog-3.34.0 | gzip > pre-receive.trufflehog-3.34.0.tar.gz
The last command will produce a compressed tar file that will need to be uploaded to GitHub Enterprise, but that will come after we explore our second option on how to create an execution environment using chroot.
chroot Execution Environment
To create chroot we will need to use debootstrap. The workflow of chroot creation looks like this:
Let’s start by creating an empty folder called chroot and then installing a base Ubuntu system using this sudo debootstrap --variant=buildd focal chroot command on the newly-created folder.
pre-receive-hook$ mkdir chroot
pre-receive-hook$ sudo debootstrap --variant=buildd focal chroot
<ommitted for brevity>
Next, we need to download trufflehog from GitHub and put it in the /bin/ folder.
pre-receive-hook$ wget https://github.com/trufflesecurity/trufflehog/releases/download/v3.34.0/trufflehog_3.34.0_linux_amd64.tar.gz
pre-receive-hook$ gunzip trufflehog_3.34.0_linux_amd64.tar.gz
pre-receive-hook$ tar -xvf trufflehog_3.34.0_linux_amd64.tar
pre-receive-hook$ sudo mv /tmp/trufflehog/trufflehog /bin/
The final step is to create a tar file. For that, change the folder to the chroot and issue this sudo tar -czvf pre-receive.trufflehog-3.34.0.tar.gz . command.
pre-receive-hook$ cd chroot
chroot$ sudo tar -czvf pre-receive.trufflehog-3.34.0.tar.gz .
<ommitted for brevity>
Uploading the Environment to GitHub Enterprise
There are two options on how to upload the tar file to GitHub. One option is to scp the file to the GitHub server and then use the ghe-hook-env-create command to add to it. Alternatively, we could use the web UI, but that needs a directly accessible link from where the GitHub server can pull the image. In both cases, we will need admin privileges. Let’s start.
pre-receive-hook$ scp -P 122 pre-receive.trufflehog-3.34.0.tar.gz admin@172.16.100.217:/home/admin
pre-receive-hook$ ssh admin@172.16.100.217:/home/admin -p 122
admin@172-16-100-217:~$ ghe-hook-env-create pre-receive-hook.trufflehog-3.34.0 ~/pre-receive.trufflehog-3.34.0.tar.gz
Pre-receive hook environment 'pre-receive-hook.trufflehog-3.34.0' (1) has been created.
Open the web browser and navigate to the GitHub Enterprise UI. In my case, it is https://172.16.100.217
In the upper right corner, click on your avatar and select the Enterprise settings option.
In the left menu, click on the Settings -> Hooks option.
Next, go to the Manage environments menu.
Finally, click on the Add environment button in the Manage pre-receive hook environments page and populate the name and URL fields.
Then click Add environment on the subsequent page to create the environment.
The browser will change to the Manage pre-receive hook environments page. Click on the Settings -> Hooks option one more time and keep the window open. We will come back to it when we are ready to create the pre-receive hooks.
Pre-Receive Hook Script
The script is what makes this whole thing come together. This is where the company can enforce the application development policy onto the developers and ensure that everything is standard across the board. For our scenario, we will use one of the examples developed by GitHub to scan the commits for certain secrets. The code looks like this:
#!/bin/bash
#
# ⚠ USE WITH CAUTION ⚠
#
# Pre-receive hook that will block any new commits that contain passwords,
# tokens, or other confidential information matched by regex
#
# More details on pre-receive hooks and how to apply them can be found on
# https://git.io/fNLf0
#
# ------------------------------------------------------------------------------
# Variables
# ------------------------------------------------------------------------------
# Count of issues found in parsing
found=0
# Define list of REGEX to be searched and blocked
regex_list=(
# block any private key file
'(\-){5}BEGIN\s?(RSA|OPENSSH|DSA|EC|PGP)?\s?PRIVATE KEY\s?(BLOCK)?(\-){5}.*'
# block AWS API Keys
'AKIA[0-9A-Z]{16}'
# block AWS Secret Access Key (TODO: adjust to not find validd Git SHA1s; false positives)
# '([^A-Za-z0-9/+=])?([A-Za-z0-9/+=]{40})([^A-Za-z0-9/+=])?'
# block confidential content
'CONFIDENTIAL'
)
# Concatenate regex_list
separator="|"
regex="$( printf "${separator}%s" "${regex_list[@]}" )"
# remove leading separator
regex="${regex:${#separator}}"
# Commit sha with all zeros
zero_commit='0000000000000000000000000000000000000000'
# ------------------------------------------------------------------------------
# Pre-receive hook
# ------------------------------------------------------------------------------
while read oldrev newrev refname; do
# # Debug payload
# echo -e "${oldrev} ${newrev} ${refname}\n"
# ----------------------------------------------------------------------------
# Get the list of all the commits
# ----------------------------------------------------------------------------
# Check if a zero sha
if [ "${oldrev}" = "${zero_commit}" ]; then
# List everything reachable from newrev but not any heads
span=`git rev-list $(git for-each-ref --format='%(refname)' refs/heads/* | sed 's/^/\^/') ${newrev}`
else
span=`git rev-list ${oldrev}..${newrev}`
fi
# ----------------------------------------------------------------------------
# Iterate over all commits in the push
# ----------------------------------------------------------------------------
for sha1 in ${span}; do
# Use extended regex to search for a match
match=`git diff-tree -r -p --no-color --no-commit-id --diff-filter=d ${sha1} | grep -nE "(${regex})"`
# Verify its not empty
if [ "${match}" != "" ]; then
# # Debug match
# echo -e "${match}\n"
found=$((${found} + 1))
fi
done
done
# ------------------------------------------------------------------------------
# Verify count of found errors
# ------------------------------------------------------------------------------
if [ ${found} -gt 0 ]; then
# Found errors, exit with error
echo "[POLICY BLOCKED] You're trying to commit a password, token, or confidential information"
exit 1
else
# No errors found, exit with success
exit 0
fi
Now we will need to commit this script to the repository which will be checked for committed secrets. The process is pretty straight forward; we will clone a repo, add a new file named pre-receive.sh containing the code above and the push it to the repository.
$ git clone git@172.16.100.217:TachTech/super-awesome-app.git
Cloning into 'super-awesome-app'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
$ cd super-awesome-app.git
super-awesome-app$ git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
pre-receive.sh
nothing added to commit but untracked files present (use "git add" to track)
super-awesome-app$ git add pre-receive.sh
super-awesome-app$ git commit -m "Script for pre-receive hook"
[main 3a9130e] Script for pre-receive hook
1 file changed, 86 insertions(+)
create mode 100644 pre-receive.sh
super-awesome-app$ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 32 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.42 KiB | 1.42 MiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To 172.16.100.217:TachTech/super-awesome-app.git
040679a..3a9130e main -> main
Now we will need to create the pre-receive hook in the browser page. Return to the web page for hooks and click on the Add pre-receive hook button.
We will name the hook Block Secrets, leave the environment as default, choose the TachTech/super-awesome-app as the repository, and select the pre-receive.sh as the script. In the enforcement section, we will deselect Enable this pre-receive hook on all repositories by default because if left enabled, this hook will be applied to all repositories in the organization and can cause damage. Finally, select the Owners can enable and disable this hook option and click the Add pre-recieve hook button to add the hook.
Great! We are almost there. Now we need to SSH to the GitHub Enterprise server, change our account to git, navigate to /data/user/git-hooks/repos/1 folder, and then change the mode of the pre-receive.sh file to executable. The whole process should happen as follows:
$ ssh admin@172.16.100.217 -p 122
admin@172-16-100-217:/$ sudo su git
git@172-16-100-217:/$ cd /data/user/git-hooks/repos/1/
git@172-16-100-217:/data/user/git-hooks/repos/1$ ls -Al
total 20
drwxr-xr-x 8 git git 4096 Jul 3 06:59 .git
-rw-r--r-- 1 git git 2904 Jul 3 06:59 pre-receive.sh
-rw-r--r-- 1 git git 38 Jul 3 06:59 README.md
git@172-16-100-217:/data/user/git-hooks/repos/1$ chmod +x pre-receive.sh
As the final step, we need to enable the hook in the repositories settings. In the browser, we will go to the repository, click Settings, and then navigate to Hooks. There we will need to enable our Block Secrets hook.
We are ready for testing! First, I will create an empty file called confidential.txt and push it to the repository. The push should go through without any issues because we are not violating the policy defined in the pre-receive hook. Let’s see how it looks.
super-awesome-app$ touch confidential.txt
super-awesome-app$ git add confidential.txt
super-awesome-app$ git commit -m "Added empty file"
[main 3874404] Added empty file
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 confidential.txt
super-awesome-app$ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 32 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 329 bytes | 329.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
To 172.16.100.217:TachTech/super-awesome-app.git
3a9130e..3874404 main -> main
super-awesome-app$
Now let’s see what happens when we add the word CONFIDENTIAL to the file and try to push it:
super-awesome-app$ echo CONFIDENTIAL >> confidential.txt
super-awesome-app$ git commit -am "Added confidential information"
[main dc95fb3] Added confidential information
1 file changed, 1 insertion(+)
super-awesome-app$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 32 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 285 bytes | 285.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
remote: pre-receive.sh: failed with exit status 1
remote: [POLICY BLOCKED] You're trying to commit a password, token, or confidential information
To 172.16.100.217:TachTech/super-awesome-app.git
! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to '172.16.100.217:TachTech/super-awesome-app.git'
As we can see from the output, the push was declined because we had something that violated the policy set by the repo administrator.
Conclusion: The Good, The Bad, and The Ugly
The Good: Centralized Enforcement and Flexibility
One of the benefits of GitHub Enterprise’s pre-receive hooks is their ability to act as a central enforcement point for developer compliance with security policies in a manner that avoids pre-commit hooks limitations. But why are pre-receive hooks not more widely adopted?
The Bad: Documentation Gaps and Limited Development
Despite their potential advantages, GitHub Enterprise’s pre-receive hooks have certain drawbacks. A key concern is the lack of comprehensive documentation and any recent enhancements, and I burned a lot of time experimenting and troubleshooting to get my environment to work.
issues like …
remote: /tmp/githook-payload.LiJP4ebq: line 17: /data/user/git-hooks/repos/1/pre-receive.sh: Permission denied
remote: /tmp/githook-payload.LiJP4ebq: line 17: /data/user/git-hooks/repos/1/pre-receive.sh: not found
… required me to root around via ssh and change permissions and paths.
The Ugly: Execution Time Constraints
A notable limitation is the hard-coded five-second execution time limit on GitHub Enterprise’s pre-receive hooks. While intended to maintain efficient processing, this constraint can potentially be restrictive for complex validation scenarios or large-scale projects - especially those with concurrent commits that trigger the pre-receive hook script.
In conclusion, GitHub Enterprise’s pre-receive hooks offer a centralized mechanism for enforcing policies and refining development practices. However, challenges related to documentation gaps, limited development, and execution time constraints underscore the importance of carefully weighing the pros and cons to determine their suitability for specific organizational needs. As with any tool, a thoughtful approach to implementation and maintenance is essential to harness the full potential of pre-receive hooks in enhancing code quality and collaboration.
Thanks for reading; I hope you liked it!
Armen
GitHub: @armartirosyan
Twitter: @armartirosyan
TachTech Engineering can help make your DevOps initiatives successful and aligned to best practices. Our sales team can help connect you to the TachTech Engineering team.
Share this on: