BioProj, Subversion and BrainVisa projects sources

The source code of BrainVISA project is on a Subversion server hosted by the BioProj platform.

See this section to access the BrainVISA sources, setup a sources client, and compile the programs.

BioProj

BioProj is the collaborative projects platform of CEA/DSV, operated by the GIPSI group. It gives access to a Subversion sources server, and to a Redmine web interface.

The web interface is reachable at https://bioproj.extra.cea.fr/redmine/. It is accessible from outside the CEA (hopefully), but the server requires an authentication: it is a secured connection, users should have an account (login and password) to access it.

To request an account creation, you should ask one of these people:

Denis Rivière
Yann Cointepas
Dominique Geffroy
Nicolas Souedet
Dimitri Papadopoulos

BioProj does not host only our BrainVisa-related projects, but also hosts several others for other teams. Users can only see and access projects they are registered in.

"Anonymous" public access

Anonymous access is not allowed on the BioProj server (due to CEA security policy). But we have a "guest" account that can be used by anyone, and has only read-only access to public projects.
We will provide the public login/password when the server is ready and running.

login: brainvisa
password: Soma2009

Bioproj web interface: Redmine

RedMine is the web interface used by BioProj. It allows several useful things:

Multiple projects management
Bug-tracking: bugs and feature requests database
Projects management: calendar, Gantt diagram, time management, virtual conference tools
Possibility to have news, documents and files
Email notification and/or RSS feed
Wiki for each project
Forums for each project
Integration with sources revision control systems: for us, SVN

More doc is available on the RedMine web site: http://www.redmine.org/wiki/redmine/Guide.

We will especially use the bug-tracking system, which allows to maintain a list of current problems, to know if a given problem has been solved, etc.
For instance: https://bioproj.extra.cea.fr/redmine/issues/300

Links are done between svn commits and bug tickets: for instance in a svn commit comment, you can write codes like:

fixes: #23
closes: #35

Such codes are interpreted by RedMine and will mofify bug tickets accordingly without the need to manually edit it on the web interface.

It is possible to create a bug ticket or a forum message via email to bioproj@cea.fr: see the BioProj FAQ.

The forums feature will perhaps be less useful for BrainVisa since we already have forums on http://brainvisa.info on a "more public" server (no login pre-required).

A few rules and conventions

We write everything in english. Sorry for the effort, but we sometimes work with foreign collaborators. We must take care that everyone can understand all messages. This is especially the rule for:
- bug reports
- comments in svn commits
- questions / answers on forums
- well, everything, in fact...
When submitting a bug report, try to be as precise as possible: choose the best matching project, the right version, etc. Bug tickets can be moved and modified afterwards so everything can be fixed, but it is just a bit more work for maintainers...
Before submitting a bug report, check / search in previous reports first if the same issue has not already been reported, to avoid duplicates.
other rules need to be written...

"BrainVisa" projects on BioProj

We now name BrainVisa the whole group of developments: the complete software environment (compilation tools, "core" projects, and toolboxes). The brainvisa executable is now part of the Axon project.
Why this naming change ?

Most users want to only know one name, for the whole thing, and BrainVisa is already the name they know.
BrainVisa is also the name of the web site which contains everything, so is already seen as the whole environment.
The name of the development project in which we develop the brainvisa application software is, after all, not so important for end users: they normally even do not need to know it, as long as the executable they run to access the user interface is still named brainvisa.

A whole set of projects have been created on BioProj to manage our developments.

brainvisa is a super-project that contains all the other ones. It will be used mainly for maintainers, for administration, and to get a list of all changes / bugs for all projects as a whole. Not everyone will get access to it since it grants access to private parts.
BrainVISA now represents the whole release (or distribution) of the different sub-projects.
Open-source core projects will be readable by everyone, and also by the [almost] anonymous user brainvisa:
- brainvisa-development: development and compilation tools, almost mandatory for all other projects
- brainvisa-communication: documentation infrastructure, web sites, bibliography database
- brainvisa-share: shared data used by several other projects
- brainvisa-installer: future project for downloader / installer program
- brainvisa-aims
- brainvisa-soma
- brainvisa-anatomist
- brainvisa-axon
- brainvisa-axon_web
- brainvisa-data_storage_client
Toolboxes will have an open-source part, and possibly a private part. Access to their RedMine project may be restricted.
Some projects have not been created yet (datamind for instance), because we are still not sure on their reorganisation. However they will be accessible via svn anyway (even if with their older perforce organization).

RedMine "roles" vs svn "groups"

Access rights and possible actions are handled differently in RedMine and subversion... One may have access to a RedMine project for bug reporting, but not necessarily to the svn repository sources, or have just a read-only access.

Subversion repository organization

Perforce archive: The former Perforce repository has been converted as is, in a "perforce" directory on the root directory of the svn server.
BrainVISA projects: New branches have been pulled from perforce archive and reorganized on svn. The new branches about BrainVISa and toolboxes development are in the "brainvisa" subdirectory. Projects have been reorganized to both match Redmine projects, and to regroup things that fit together.
- In each project (or sub-project), there are sub-directories corresponding to branches:
  - New developments are done in the "trunk" branch: trunk directory
  - Stable branches are in branches/<branch-number> (one directory for each numbered branch)
  - Additionally, there is a tags directory at the same level as trunk and branches, which contains, like branches, a subdirectory for numbered versions. These versions are not full branches, but fixed "tagged" versions at a given date, and do not change any longer. These tags are used for final releases with full version numbers (eg. 3.1.7), while branches continue to get bug fixes, and are not numbered with the tiny number (3.1 only). In other words, tags are static snapshots of the corresponding stable branch at the moment of an official release.
- Subversion does not have user-side "client" definitions like Perforce. However most developers do not want to checkout the entire repository with all branches. To deal with the branches / client problem, we also have setup a "brainvisa/source_views" directory, which contains some pre-organized groups of projects on a given version/branch. They are defined as svn "externals" (links to several other subdirectories). Normally people will just use these source_views since they are already setup with what most people expect: the development branch, or the stable branch, of all main projects. In this source_views directory, we have two levels of subdirectories:
  - group of projects:
    - opensource for open-source projects, publicly readable
    - standard for projects that most people developing within our framework will need/want to access.
    - all all projects (both public and private): will be accessible only to administrators and people who have access to all sources
    - possibly other projects groups for extra non-standard projects
  - branch: normally in each of the above directory, there is a trunk and a stable subdirectory for (respectively) the main development branch and the "stable" bugfix branch.
  It is possible to checkout one of these projects/version directory, for instance brainvisa/source_views/standard/trunk: svn co https://bioproj.extra.cea.fr/neurosvn/brainvisa/source_views/standard/trunk
Personal projects: Other projects that are not linked to BrainVISA development and personal projects have been put in a perso directory.

New organization details

BrainVISA CORE

Bioproj	Svn				Description
	/--	brainvisa
brainvisa-development	\|	\|--	development		Tools for software development in BrainVISA project. Contains brainvisa-system for installation of third party software on several OS and brainvisa-cmake for compiling and packaging with CMake.
	\|	\|	\|--	brainvisa-cmake
	\|	\|	\|--	brainvisa-svn
	\|	\|	\|--	brainvisa-system
	\|	\|	\|--	brainvisa-release
	\|	\|	`--	build-config
brainvisa-installer	\|	\|--	brainvisa-installer		Software for easily download and install all BrainVISA compatible packages. For a new BrainVISA user, it will be first element to download and to execute. All other packages will be installed with BrainVISA-installer. It will also provide the possibility to automatically download packages. It is still work in progress.
brainvisa-share	\|	\|--	brainvisa-share		Files and data that are shared among several projects.
brainvisa-soma	\|	\|--	soma		Contains general purpose packages that are necessary for Axon but can be used in other projects. Soma-base contains Python modules (that corresponds to soma-infra project under Perforce). Soma-io will be the shared input/output C++/Python library. Soma-qtgui contains all Qt related graphical interface tools.
	\|	\|	\|--	soma-base
	\|	\|	\|--	soma-io
	\|	\|	`--	soma-qtgui
	\|	\|	`--	soma-workflow
brainvisa-aims	\|	\|--	aims		C++/Python image processing and general purpose library. Contains (and hide) many Perforce projects (graph, cartobase, cartodata, aimsalgo, aimsdata, ecat, ecat+, vidaIO and pyaims).
	\|	\|	\|--	aims-free
	\|	\|	`--	aims-gpl
brainvisa-anatomist	\|	\|--	anatomist		Interactive visualization of multimodal data and for manipulation of structured 3D objects. Contains anatomist and pyanatomist.
	\|	\|	\|--	anatomist-free
	\|	\|	`--	anatomist-gpl
brainvisa-axon	\|	\|--	axon		Former brainvisa Perforce project.
	\|	\|--	communication		Management of website, documentation and publications.
	\|	\|	\|--	bibliography
	\|	\|	\|--	documentation
	\|	\|	\|--	latex
	\|	\|	`--	web

BrainVISA TOOLBOXES

Redmine	Svn				Description
	/--	brainvisa
brainvisa-axon_web	\|	\|--	axon_web		Web server application to access Axon processes through a web browser (former brainvisaweb).
brainvisa-brainrat	\|	\|--	brainrat
	\|	\|	\|--	bioprocessing
	\|	\|	\|--	brainrat-gpl
	\|	\|	`--	brainrat-private
brainvisa-connectomist	\|	\|--	connectomist
	\|	\|	\|--	connectomist-gpl
	\|	\|	`--	connectomist-private
	\|	\|	\|--	old_connectomist-gpl
	\|	\|	`--	old_connectomist-private
brainvisa-cortical_surface	\|	\|--	cortical_surface
	\|	\|	\|--	cortical_surface-gpl
	\|	\|	`--	cortical_surface-private
	\|	\|	`--	freesurfer_plugin
brainvisa-data_storage_client	\|	\|--	data_storage_client		Access to data acquired on scanners (former NMRServer).
	\|	\|--	datamind
brainvisa-fmri	\|	\|--	fmri
	\|	\|	\|--	fmri-gpl
	\|	\|	`--	fmri-private
	\|	\|--	nuclear_imaging
	\|	\|	\|--	nuclear_imaging-gpl
	\|	\|	`--	nuclear_imaging-private
brainvisa-nuclear_processing	\|	\|--	nuclear_processing
	\|	\|	\|--	nuclear_processing-gpl
	\|	\|	`--	nuclear_processing-private
	\|	\|--	optical_imaging
	\|	\|	\|--	optical_imaging-gpl
	\|	\|	`--	optical_imaging-private
brainvisa-fmri	\|	\|--	pyhrf
	\|	\|	\|--	pyhrf-gpl
	\|	\|	`--	pyhrf-free
brainvisa-ptk	\|	\|--	ptk		C++ image processing and general purpose library (former NMR).
brainvisa-sulci	\|	\|--	sulci
	\|	\|	\|--	sulci-data
	\|	\|	\|--	sulci-gpl
	\|	\|	`--	sulci-private
brainvisa-t1mri	\|	\|--	t1mri
	\|	\|	\|--	t1mri-gpl
	\|	\|	`--	t1mri-private

Other directories

Svn					Description
/--	brainvisa
\|	\|--	source_views			Client views to main projects and current versions, to allow easy checkout and workspace for most users
\|	\|	\|--	all		All projects - useable only by administrators
\|	\|	\|	\|--	trunk	trunk branch of all projects
\|	\|	\|	`--	stable	stable branch
\|	\|	\|--	opensource		Open-source projects - the "brainvisa" semi-anonymous user can checkout these directories.
\|	\|	\|	\|--	trunk	trunk branch
\|	\|	\|	`--	stable	stable branch
\|	\|	\|--	standard		Strandard projects - used by most developers
\|	\|	\|	\|--	trunk	trunk branch of standard projects
\|	\|	\|	`--	stable	stable branch
\|	\|	\|--	standard_ptk		Strandard projects + private project PTK (example)
\|	\|	\|	\|--	trunk	trunk branch
\|	\|	\|	`--	stable	stable branch
\|	\|	`--	...
\|	`--	releases			views to "official" BrainVISA releases, containing a specific set of projects, each with a specific version / tag
\|		\|--	3.2.4		an example of numbered BrainVISA release
\|		`--	...
`--	perforce				Old Perforce repository imported on svn, exactly as it formerly was on the Perforce repository. It will not be used any longer for projects which now have a svn branch in the `brainvisa` directory. Some old projects that have not been copied in the new branches will be still accessible via this perforce repository archive.
`--	perso				Personal projects and other projects not directly linked to BrainVISA development.
\|	\|--	name			Personal project
\|	\|--	distance
\|	\|--	hrf
\|	\|--	INRIAlign
\|	\|--	medinria
\|	\|--	fmridocs
\|	\|--	parallel
\|	\|--	pyhrfdocs
\|	\|--	servier

If you have difficulties to find a project in the new organization, have a look to this wiki page on bioproj server. There is a list of the old perforce projects and their new place in svn.

Subversion use

The use of "subversion" is quite the same as "perforce" use. It is based upon a main command line which is "svn". Many graphical interfaces to use subversion also exist. "kdesvn" is one of these.

To get all functionalities, especially on branching/merging, you should ensure to use subversion 1.6 or newer. An older client (1.4 for instance) will be OK for most common functionalities, but will miss a few things.

Identification

To download files from a svn repository, you will need a valid user and password, they are the same as the ones used to connect to Bioproj. The public account brainvisa can be used to access the subversion repository but will provide a read only access to open source projects.

Configuration file

At first use of subversion, a directory named ".subversion" is created in your home directory. It contains a configuration file "~/.subversion/config". This file can be used to manage svn client configuration. It is organized using sections that contains key and value pairs.
As an example, it allows to enable/disable the password storage or to specify a different text editor than the default used.

[auth]
store-passwords = yes
store-auth-creds = yes

[helpers]
editor-cmd = kwrite

Note that the "store-auth-creds = yes" option will allow svn to store your passwords in your ~/.svn/ personal directory, but will prevent it from asking you a password for each access to the server, so it is convenient.

Svn does not have standard tools to visualize differences and merge files. It is necessary to use external tools. These can be :

kompare (visualize file differences)
kdiff3 (merge files)

These tools have to be set in the [helpers] section of the configuration file "~/.subversion/config" :

[helpers]
diff-cmd = kompare_svn
diff3-cmd = kdiff3_svn
merge-tool-cmd = kdiff3_svn

kompare_svn and kdiff3_svn are two little scripts that just call kompare and kdiff3 with the arguments in the correct order, since svn does not use the same as these commands. The two scripts can be found in the svn repository, in development/brainvisa-svn/trunk/bin/.

Svn documentation

A quick reference card can be downloaded here and a complete documentation is also available at http://svnbook.red-bean.com.

Svn equivalents to Perforce commands

p4 command	svn command	comment
`p4 help`	`svn help`	get commands list
`p4 sync`	1st time: `svn co <URL>` then: `svn up`	or `svn checkout`
`p4 open`, `p4 edit`	nothing	files are directly editable
`p4 revert`	`svn revert`
`p4 add`	`svn add <directory>`:	adds a directory and what it recursively contains
	`svn add <file>`:	adds a single file Note that unlike with perforce, directories must be added in svn.
`p4 opened`	`svn st`	or `svn status`
`p4 submit`	`svn ci`	or `svn commit`
`p4 diff`	`svn diff`
`p4 delete`	`svn delete`
`p4 changes`	`svn log`	shows revisions involving the current directory
`p4 resolve`	`svn update` -> 3 files are created when there is a conflict: `filename.mine`, `filename.r<oldrev>`, `filename.r<newrev>` `svn resolved`: after having resolved the conflict
`p4 integrate`	`svn cp <src> <dst>`	or `svn copy`. `src` and `dst` may be server-side URLs, and may be used for branches creation.
to integrate branches:	`svn merge`	To report changes from one branch to another one: several arguments are possible to compare two branches and apply changes to the current repository copy. Two branches revisions may be specified: `-r N:M <source_url>` Two branches may be specified: `<url1>@<rev1> <url2>@<rev2>` To report all changes done in a branch into the trunk branch: cd project/trunk svn merge -r <branch creation revision>:<end branch revision> <branch url>
`p4 integrate` + `p4 delete`	`svn mv`	or `svn move`
`p4 client`	no equivalent	we use externals on server side

or see http://blogs.sun.com/owenkellett/entry/migration:_perforce_to_subversion.

Using BrainVISA sources

First, don't try to checkout the entire repository:

it is _huge_ and contains many versions of all projects
you will probably not have the needed access rights

We developped a script named bv_maker in order to easily create a complete source synchronization and compilation pipeline for a set of Brainvisa projects. This part will describe how to use this tool to get and compile the source of a set of selected Brainvisa projects. A more detailled documentation can be found in BrainVISA Bioproj Wiki.

Getting bv_maker

bv_maker is part of the brainvisa-cmake project. Since you need bv_maker to download the sources and do the first build directory, you may have to download a temporary version with the following code:

For Linux and MacOS:

svn export https://bioproj.extra.cea.fr/neurosvn/brainvisa/development/brainvisa-cmake/trunk /tmp/brainvisa-cmake
cd /tmp/brainvisa-cmake
cmake -DCMAKE_INSTALL_PREFIX=. .
make install

For Windows:

svn export https://bioproj.extra.cea.fr/neurosvn/brainvisa/development/brainvisa-cmake/trunk /c/tmp/brainvisa-cmake
cd /c/tmp/brainvisa-cmake
cmake -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=. .
make install

You can now use /tmp/brainvisa-cmake/bin/bv_maker to setup your environement.

Getting the sources

First, you have to edit a configuration file to define your source and build directories and which projects you want to get. This configuration file must be written in $HOME/.brainvisa/bv_maker.cfg.

Typical configuration:

# definition of the source directory : standard projects in trunk version except web project because it takes space 
[ source $HOME/brainvisa/source ]
  + standard trunk
  - web

# definition of the build directory : build standard projects from the source directory except connectomist-gpl and connectomist-private components 
[ build $HOME/brainvisa/build/trunk ]
  make_options = -j4
  standard trunk $HOME/brainvisa/source
  - connectomist-*

Then, you just have to run the following command to download the sources into the source directory defined in the configuration file:

/tmp/brainvisa-cmake/bin/bv_maker sources

Compiling

We now use CMake, a cross-platform and open-source build system to build our projects. There are 2 steps :

configure: Makefiles are generated in build directory from the CMakeLists.txt files that are located in source directory
build: make command is executed in build directory and all the projects are compiled.

Type the following commands to configure and build the sources:

/tmp/brainvisa-cmake/bin/bv_maker configure
/tmp/brainvisa-cmake/bin/bv_maker build

After this step, you have a version of brainvisa-cmake installed in each build directory you have defined. You can therefore find bv_maker in /bin/bv_maker. So you can delete the temporary installation of brainvisa-cmake:

rm -Rf /tmp/brainvisa-cmake

Running programs

Once compiled, everything will get "installed" in <build_directory>. You must setup your PATH variable to include its bin/ sub-directory, your LD_LIBRARY_PATH variable to include its lib sub-directory, your PYTHONPATH to include its python sub-directory, and setup a BRAINVISA_SHARE to point to its share sub-directory.

To make it easier, we provide a program that sets up all the needed environment variable : bv_env. You can find it in your build directory in subdirectory bin:

. <build_directory>/bin/bv_env <build_directory>

Then you can run the programs: anatomist, brainvisa, AimsFileInfo, bv_maker, etc.

Build documentation (docbook, doxygen, epydoc)

bv_maker doc

2011/03/30