DevelopmentEnvironmentIdeas

As we get more projects, and more developers, it becomes harder to manage what we're doing. Here are some ideas for making this easier.

Requirements

• Accessible to both Windows and Unix desktop-using developers.

• Any clients easy to install, or already normally installed.

• Can get going coding and testing very quickly.

• Low administrative burden for adding new developers (we want lots of them!)

• Privileges separate for altering/editing each project

• Privileges separate for accessing live databases with personal info in them

• Easy deployment of new live version

• Record what version was last deployed, and be able to go back to previous versions

• Ability to enforce automatic tests before deployment

Ideas

• Have lots of separate sandboxes, such as sams.notapathetic.com is at the moment

• Generate httpd.conf and postgres conf files from developer and project definition data (probably a database is overkill, maybe a text file, Perl script with hashes in it, whatever)

• Use WebDav to let people edit their sandbox from Mac/Windows if they don't like Unix command line editors http://www.serverwatch.com/tutorials/article.php/2176771 http://www.swissdisk.com/index.html

• Web interface for deploying, and recording deployments, and rolling back

• Staging/sandbox sites checked out from CVS.

• Maybe live site also a CVS checkout for easy but really quick important fixes.

• Web interface for editing text files (e.g. email templates, simple HTML text) on sites for the likes of Tomski. Or maybe WebDav gets us this somehow?

• Have each site self contained, so can be more easily moved between servers, run separately. In particular, be very careful about compatibility of any shared service (MaPit, EvEl).

Other Stuff

• Some way of giving users stuff to do without having to give them access. Makes it easy to respond to an offer of help, with minimal time cost if that person then doesn't do anything much.

Brief comments

An effective way to sandbox developers' sites should be to run multiple apache instances (rather than have lots of vhosts in apache) and then use a front-end proxy (squid, probably) to direct web requests to these. Each sandbox site runs under one UID and listens on one private IP address; we create a user account for each such UID and give access to it (SSH keys, shared passwords, sudo, ...?) to everyone working on this site. They can then do whatever they need to with that particular apache instance without either breaking other sites or suffering complicated restrictions from vhost setups.

I propose we then have a standard directory structure for each such site (probably generalising /data/vhost/* from very) and write a script to start all the apache instances under appropriate UIDs at system start; after that point it's up to the individual users to keep apache instances running.

Access to databases:

The concern here is to protect any personal data (as processed on production sites) from leaking out to people who should not have access to it. If we are going to be mixing development and production sites on single machines (there are sound reasons we might want to do this, and I suspect it'll happen anyway by default) we have two basic approaches: different database installations for production and development use; or rely on the database's own permissions model. Our unofficial policy has been to use Postgres for everything and to use Postgres's mapping of UNIX to database users to control access to the database; we can continue this policy to separate development from production use of the database.

Remote filesystem access

The argument here is that some developers are not facile with UNIX but we obviously don't want to reject their contributions solely on this basis. One way to do this is to let such volunteers edit files directly from their MS Windows etc. desktop machines. There are lots of thorny issues here which I'm going to ignore for the moment.

Basically the choices here are WebDAV or something SMB-like. I've never come away from looking at this area without concluding that it's a complete trainwreck and that's still my impression now. The problem with SMB is that it isn't safe for use over a public network so we would have to encapsulate traffic in some other protocol (typically one uses VPNs for this, but probably SSH tunnelling could be made to work) but that would of course mean having to support whatever client software is needed to make it work. mod_dav or another WebDAV implementation (there are some CGI ones) therefore seems the inevitable solution despite its numerous problems.

We can create a templated configuration for mod_dav for each developer sandbox so that developers can set this up for their own sandboxes; obviously any such access should be over a secured link, but this can fairly easily be accomplished via secure.mysociety.org and some other proxying hack. Basically, however much this sucks, I think we should just bite the bullet and go for WebDAV. (If nothing else it will be interesting to predict what sorts of support problems this generates for us.)