Commit Watcher finds interesting and potentially hazardous commits in git projects. Watch your own projects to make sure you didn't accidentally leak your AWS keys or other credentials, and watch open-source projects you use to find undisclosed security vulnerabilities and patches.

At SourceClear, we want to help you use open-source software safely. Oftentimes when a security vulnerability is discovered and fixed in an open-source project, there isn't a public disclosure about it. In part, this is because the CVE process is onerous and labor intensive, and notifying all the users of a project isn't possible.

Oh, and about that UI. Commit Watcher is intended to be an API accessible backend service. The UI is only there for testing, and the scope of functionality is limited to collecting commits and auditing them against a set of rules.

Check out the dozens of rules and patterns in the srcclr/commit-watcher-rules repository that help find leaked credentials and potential security issues. Just open an issue or PR in that repo if there's a rule you'd like to see added.

Additionally, if you find a security issue on an open-source project using Commit Watcher, our security research team would love to help verify it. You can open an issue against this repo from the UI, or just drop a link to the offending commit in a new issue.

Install and configure Ruby using RVM or Rbenv. Avoid using the system's bundled Ruby to avoid permission issues during installation/setup.

Install MySQL and Redis. On Mac, with Brew, you can do that with this command:

Follow the instructions Brew gives you so the services are started properly.

Install gem dependencies:

Then setup some Rails secrets and passwords:

The rest of the setup depends on how you want to run Commit Watcher. You can either run it locally, which is good for quick development, or you can run it with Docker.

To use email notifications, set your Gmail username and password with these commands:

If you'd like to use another email provider other than Gmail, you'll have to change these two files: config/environments/development.rb and config/environments/production.rb.

Create the database, load the schema, and seed it with some sample rules:

Now you're ready to start Rails with:

To start processing jobs, in another terminal:

First, change the root and user passwords in .env.db.

Second, modify config/database.yml by commenting out socket in favor of host, like this:

Alternatively, for RDS, setup the external RDS URL:

Then, modify config/database.yml by commenting out socket in favor of host, like this:

And modify docker-compose.yml by commenting out - db in the web: and sidekiq: sections, like this:

Now start everything going with:

This downloads the images and builds the database and rails app containers. When it's finished building, and both containers are running, you should see rails messages like this:

Stop Docker with Ctrl+C so the database can be setup with:

Now start everything up again with:

If using Docker, the server will be accessible from the IP address given by:

To crawl any projects, you must set a GitHub API token in the default configuration. This can be reached here: http://localhost:3000/configurations/1/edit.

The web UI contains a dashboard which links to all available pages. It's located here: http://localhost:3000/.

Sidekiq dashboard is here: http://localhost:3000/sidekiq/cron.

The process starts by every few minutes any project which hasn't been checked in a while is polled for new commits. These commits are then checked against whatever rules are setup for the project. Any commits which match are recorded and available at the /commits endpoint.

Everything is broken up into different Sidekiq jobs. There are three:

1. Selecting projects which need to be polled
2. Collecting new commits
3. Auditing a single commit

The API endpoints are similar to the web UI and are documented by code.

The app must have a hostname to access the API endpoints. This can be done in development by adding a record to the host file:

Then the API can be accessed by:

Rule types are defined and described in config/rule_types.yml. They are:

• filename_pattern - Regular expression for a filename
• changed_code_pattern - Regular expression for a changed line
• code_pattern - Regular expression for any code in a changed file
• message_pattern - Regular expression for a commit message
• author_pattern - Regular expression for a commit author name, normalized to "name "
• commit_pattern - Combination of code_pattern and message_pattern
• expression - Boolean expression referencing one or more rules

This is a special rule type that allows for combining multiple rules in a boolean expression. The boolean expression has three operators: && (and), || (or), ! (not), and also allows for parenthetical expressions.

For example, if there are three rules:

1. is_txt - /\.txt\z/ (filename_pattern)
2. has_lulz_msg - /\blulz\b/ (message_pattern)
3. has_42 - /\b42\b/ (code_pattern)

To create an expression rule which would match commits that include "lulz" in the commit message and contains at least a single text file or has a file with the word "42":

To match a commit where any file is not a text file and includes "42":

Automated identification of security issues from commit messages and bug reports, FSE 2017