Overview

Background

Developers need a place to work. The immediately easiest way to do this is by editing code on the production site. This is of course very dangerous, but it's tempting, and for a new site, there is no production site yet so people often don't think about the future pain the habits will cause.

Next in the natural progression of things seems to be having "a development site". That gets out of date easily, and multiple developers or even a single developer with mulitple simultaneous projects, steps on each others' toes. It becomes a mess quickly.

Version control is needed to keep a history of changes, so that older code can easily be recovered, and to assist in documenting who did what. Version control systems can also be used to push code between production and development sites.

At End Point, we have gone further and developed a development system we call "camps". The system was developed first at Backcountry.com and assumed most of its current form for CityPass. It is the result of lots of small improvements over the years, leading from a simple "production" vs. "development" setup that didn't scale at all, to the complex but very helpful systems we have now.

High-level concepts

Need to keep engineer-written code (including JavaScript, CSS, etc.) separate from:
- Automatically-generated files (sessions, temporary, timed-build, logs, etc.)
- Customer user-controlled changes (site content, settings, etc.)
Files should be stored in version control (including DDL changes).
Files are authoritative in version control, database is authoritative in production database. That means:
- Database should contain all "settings" or "variables" that web admin users would change.
- Database should contain nothing hardcoded for the particular paths or URLs of the site.
- Database sequences should be used for order numbers, etc., not Interchange counter files.
Need to be able to work on multiple projects simultaneously without affecting each other.
Should be no connection between development and production environments except through commits.
Development environments should be as close as possible to production environment in all ways.
Should promote easy peer code-reading as you go along, without fuss.
Needs to be easy to use for developers and our clients who test our code.

User-level features

We call the development environments "camps". This is much easier to use in conversation and more specific than "development environment".
Have a camps index page, giving lists of all camps, links to each part of each camp, notes and owner per camp, etc.
A single engineer generally has multiple camps for simultaneous projects.
Camps are numbered 0-99 so you can build them and throw them away as you need.
Camps 0-9 are special, generally reserved for staging and test environments:
- Code there is expected to be as close to production as possible, except with new code that should be pushed out very soon.
- They only contains code already checked into version control.
- The client tests code there one last time before rollout.
- They allow testing simultaneous changes other developers have made, and rolling out batches of changes together.

Technical details

Have separate HTTP and HTTPS (with separate SSL certificates) just like production site.
Have their own subdomain so cookies don't affect live site or other development environments.
Each development site has its own:
- Apache instance.
- Interchange instance.
- Database instance.
Have predictable nonstandard ports in their URLs, database listeners, etc. to ensure uniqueness.
Code stored in version control (and docroot too wherever possible):
- Git and Subversion (optionally with the SVK client) are available now; others not hard to integrate if required.
- Automatic email of commit notices with inline diff of changes.
- Version control repository has a web interface for browsing/accessing.
Scripted build of new camps, and destruction of old ones.
Development databases can be updated independently from the production database, to stay current.
The staging environments gets code only from version control.
The production environment is a version-controlled camp just like the other environments, so it's easy to diff and see if any local changes were made, apply updates, etc.

Camp tools

Custom programs

The ~camp/bin directory contains programs for dealing with camps. We are trying to keep the programs and options as similar as possible between different clients, but there will likely be a few differences you'll encounter.

The programs are:

mkcamp

Use this to create a new camp for yourself. Various arguments are required:

--comment argument is required and allows you to explain what this particular camp is for. That's useful when you have multiple camps, and not very useful if you only have one.
--type=$type allows you to specify the type of camp; this is relevant on multi-camp systems, and does not apply (currently) to single-project systems. The default and currently only available option is 'basic'

Also of interest:

--vcs=(svn|svk) allows the specification of the version control system to use for the new camp. The default is svn.

re

The re program is for restarting the web server, database server, and/or application server for your camp. To restart the application server only when you're in a camp working directory, it's simply:

If you want to deal with a different camp or aren't in a working directory:

re $number

To restart web server, database server, and application server, in the right order:

re --all $number

There are also --stop and --start arguments. The default is --restart.

refresh-camp

General utility for refreshing your camp from some canonical source, which varies depending on the type of refresh you perform. Currently, refreshing can be any combination of:

--db: rebuild your camp database from the source database -- --specified in the skeleton for the camp type
--vcs: update your working copy from the source repository in a -- --manner appropriate to your chosen version control system (svn -- --versus svk)
--files: update your non-version files from their sources as -- --specified in the camp type's copy-paths.yml file.

Note that refreshing your camp database will likely result in your passwords changing. You'll need to check your relevant files (like catalog_local.cfg in IC or database.yml in Rails) to update them with the new password. If the camp uses Postgres, then your .pgpass file will have been updated. If using MySQL, the camp's mysql.yml file will have been updated. You can find the new passwords from the appropriate source.

rmcamp

Use this program to permanently delete all parts of a camp: web server, database, and application. Its syntax is a little different to make it less likely you'll invoke it by accident:

rmcamp --number=$number

camp_change_comment

This will allow a user to change the comment of a camp.

camp_change_comment --camp=$camp_number --comment=$comment

Database client wrappers

The camp system provides database client wrappers for the databases it supports. These wrappers facility ease of use of your camp's database, eliminating the need for port specification, password specification, etc.

The default user and database to which these connect when no arguments are provided are determined in the camp type's configuration files, by the db_default_database and db_default_user entries.

mysql_camp

This is a shortcut to get into your camp database without having to type the port, user, and database every time. If your current working directory is inside one of your camps, you can simple type:

mysql_camp

and you'll be in the right database. If you want to use a different camp or are not in a camp directory:

mysql_camp $number

It also passed on any additional arguments to psql directly, so you can do e.g.:

mysql_camp 21 -e 'select * from products'

Given the lack of flexibility and irritating format of my.cnf, the camp system uses a custom YAML file per camp to hold camp-specific settings (like database names, user/password combinations, etc.). The default user and database used by the mysql_camp utility are determined by the entries in the mysql.yml file within the camp.

psql_camp

Exactly analogous to psql_camp, except it wraps the MySQL client instead of the Postgres client.

Standard programs

svn

You'll use the svn commands in the usual way. For example, to bring the latest changes from the Subversion repository into your camp:

cd ~/camp$number svn up

To see what files you've changed in your working copy:

svn stat

To see exactly what you changed in each file:

svn diff

To undo your changes, either delete the file and re-update, or:

svn revert

To commit your changes, make sure first to update, then diff and double-check your changes. Then:

svn ci

That will open a text editor so you can type a commit message. If you abort the edit without saving, Subversion will ask whether you want to abort the commit. So it's not too late then to stop.

The Subversion command-line help is quite good, so try svn help for a reminder of the options, and see our Subversion page for more details and offsite links. Most important aside from the built-in help is the Subversion book.

grep

Using grep to search through code is a bit of a pain because of all the false positives in the .svn directories. One way to exclude those is:

grep -ri 'your search text' . | grep -v '/\.svn/'

Camp work

Process

After committing a change to the repository, you can test further in several ways:

Keep a pristine camp for yourself where you can apply updates and test them apart from any other changes you're working on.
Have a co-worker apply your change to his/her camp and test it for you.
Update camp0 and do a final test there.

Then, once you're ready, coordinate with other developers on the project and with the client to roll out changes to the live site.

Applying changes at any location generally involves:

Updating the Subversion working copy.
Applying any new DDL scripts in the migrations directory.
Restarting Interchange if any pre-compiled code changed or was added or removed.
Test the function you changed.
Test a few common site paths, such as home page to product to cart to checkout.
In all but the simplest changes or docroot changes only, it's wise to place a test order.

Camps make it easier to work together, but no technological solution is a substitute for communication with the various stakeholders.

Reboots

We do not automatically start up camps when a server boots. Camps must be started manually. This is because generally camps being down for a while is no big deal, and starting all the Apache, MySQL, and Interchange daemons for all camps at boot time puts a lot of strain on a server that's already trying to start lots of other things. Also, camps may go unused for days or weeks, and it's best to stop them when they're not in use to free up memory and CPU.

Problems

The Konqueror browser's security policy does not share cookies between nonstandard ports on the same hostname, so Konqueror can't be used for end-to-end testing in a camp. For example, going from http://mysite/ to https://mysite/ carries over cookies. But going from http://mysite:9000/ to https://mysite:9100/ does not, in Konqueror. All other browsers seem to work fine here, though.