We need a CPAN/PyPi/easy_install kind of tool for Bro that makes it easy to both provide and install custom 3rd party scripts. The tool would track dependencies with Bro versions and other scripts; provide for easy upgrades to new versions of a script; and potentially help with configuration and/or doc generation.
On the bro.org server side, a web interface for browsing available 3rd party scripts would be very useful.
github‘s popular fork-pull-merge model is an excellent fit for sharing community repositories. (We could use https://github.com/Bro as base for officially supported community scripts.) Users that want to extend the official scripts simply perform the changes, open a pull request, and we can review it. The same model works, of course, for user repositories. The nice thing is that people can comment on it and follow-up changes are nicely reflected. Moreover, github automatically lists contributors to a certain file, so this encourages coders to share, participate, and get “reputation”–akin to stackoverflow.
Unlike homebrew, it may not make sense for Bro to have single huge repository that contains all scripts and that users continuously branch from. Bro users may want to organize their scripts into different repositories and combine multiple repositories ad libitum, because some repositories may contain sensitive elements and should not be public. Yet, private repositories can still be easily integrated with this new framework.
Having separate repositories also keeps the repository size under control. Although the input framework will be the main mechanism to source external data, I could imagine some folks would like to stuff some file-based intelligence in their repositories.
However, separate repositories require integration efforts, but this is manageable in my eyes and outlined below.
Directory structure: the directory structure provides implicit naming and some meta data. If the directly structure is violated, Bro will not be able to automatically load scripts. These are the rules:
Sub-directories in the repository represent “addressable units,” e.g., names that users can refer to. I call these from now on a rebro. Each rebro must have at least one file called __load__.bro. This is the entry point for Bro.
Example: (base = https://www.github.com/joe-user/bromising/)base/ base/a/ base/a/__load__.bro base/a/aa.bro base/a/ab.bro base/b/__load__.bro/ base/b/x/__load__.bro base/b/x/xx1.bro base/b/x/xx2.bro base/b/y/yy1.bro
In this example, we have the following rebros:repo/a repo/b repo/b/x
By placing __load__.bro files judiciously, users can explicitly decide what parts of the repo may be interpreted as a rebro. We may also allow putting a __load__.bro file in the top-level directory.
Each __load__.bro file contains additional meta data. Here is an example:##! A short paragraph that describes this rebro at the ##! beginning of the file. ##! ##! @version: 0.42 ##! @author: Joe Bro <firstname.lastname@example.org> ##! ##! @license: BSD-style (should be the default unless specified) ##! ##! @bro: 2.1 ##! @dependencies: b/x, https://my-repo.git/foo ##! ##! @tags: tls, x509, browser
The @-prefix inside comments may be used as a specific marker for parsing. But any other character would do as well.
@author: An optional field that identifies the script author’s name and email address. If not given and the author hosts its repository on github, we may extract this data from github.
@license: An optional field to name the license under which this rebro should go
@bro: A mandatory field that specifies the Bro version this rebro is compatible with. Examples:
exact version match, e.g., =2.1
minimum version, e.g., >=2.0
range of comptabile versions, e.g., 2.0-2.1
list of compatible versions, e.g., 1.5, 2.0beta
@dependencies: An optional field of internal and external rebro dependencies. Internal dependencies are specified relative to the repository top-level directory and external dependencies need a full-qualified rebro path.
@tags: An optional field that imposes a logical structure over rebros. Or we just keep it an unordered list of tags that users can combine at will:@tags: tls, x509, browser
would associate three tags with this rebro. We may pre-define a set of categories under which users can label their rebros:@tags: webapp/facebook, webapp/facebook would categorize the rebro hierarchically.
Instead of having the meta data written into __load.bro, it may be better to put it into a README file in the same directory, because there exists already infrastructure to pick up README files in reST format to generate documentation.
Let the union of all rebros be the universe. We, the Bro team, should provide a global view of the universe by encouraging users to register their rebro’s with us. The point of the universe is that it allows a global search across all existing rebros out there. One should be able to search the universe by tags, names, versions, etc.
We may cache this meta data in a separate repository. A script may regenerate the meta data whenever a new rebro is added to it.
Configuration: Bro needs to be told which script repositories to integrate. I suggest all repositories sit in a one directory, that is controllable via BroControl.cfg:
Community = $PREFIX/share/bro/community
Inside that directory, we have two subdirectories: one for all the repositories and one for the universe:
$Community/repositories/git.bro.org/skype-analyzer $Community/repositories/github.com/git-user/experimental-scripts $Community/repositories/github.com/git-user/popular-stuff $Community/universe/...
One should be able to use BroControl to add new repositories at runtime and reload Bro with these scripts. Here are some examples that I will explain below:
> rebro add https://git.bro.org/skype-analyzer [alias] > rebro add git-user/experimental-scripts [testing] > rebro rm testing > rebro load alias/decryptor > rebro load -r alias/spit-injector > rebro list > rebro list -u > rebro list -u -t > rebro pull > rebro deps alias > rebro deps alias/decryptor > rebro license alias
add: downloads a remote repository and register make all rebros inside it loadable
rm: removes a remote repository.
load: loads a specific rebro. The command fails unless all dependency requirements are met:--recursive|-r: downloads dependencies automatically
list: lists all available loadable rebros:--universe|-u: performs a global search in the universe --tags|-t: lists rebro's by tags
deps: lists dependencies of a rebro.
license: displays licencing information about a rebro.