g5k-campaign

A tool to launch campaigns on Grid'5000, based on the concept of extensible campaign engines.

Installation

  > gem install g5k-campaign --source http://g5k-campaign.gforge.inria.fr/pkg --source https://rubygems.org

If you are installing from within Grid'5000, add the proxy (-p) flag:

  > gem install g5k-campaign --source http://g5k-campaign.gforge.inria.fr/pkg --source https://rubygems.org -p http://proxy:3128

Prerequisites

This tool uses the Restfully http://github.com/crohr/restfully ruby gem to connect to the Grid'5000 API. You MUST set up a specific configuration file on the machine from where you will run g5k-campaign commands. The recommended location for this file is in ~/.restfully/api.grid5000.fr.yml. You can generate one using:

  > mkdir ~/.restfully
  > echo "
  username: $USER
  password: GRID5000PASSWORD
  base_uri: https://api.grid5000.fr/stable/grid5000
  " > ~/.restfully/api.grid5000.fr.yml
  > chmod 0600 ~/.restfully/api.grid5000.fr.yml

WARNING: Within Grid'5000, the environment is unsure. So since you are automatically authenticated, you MUST NOT put your Grid'5000 credentials in the configuration file. If you are using the tool within Grid'5000, the command to enter would look like:

  > echo "
  base_uri: https://api.grid5000.fr/stable/grid5000
  " > ~/.restfully/api.grid5000.fr.yml

Usage

  > g5k-campaign -h
    * Description
      g5k-campaign: Launch campaigns on Grid'5000.
                    Nodes that were successfully configured will be displayed on STDOUT.
    * Usage
      g5k-campaign [-i engine] [options] [EngineName]
    * Options
        -a, --authorized-keys-file=      Specify an authorized_keys file to use instead of your public key [default=/home/sbadia/.ssh/id_dsa_g5kadmin.pub].
        -c, --config=                    Pass the Restfully YAML configuration file. See <https://github.com/crohr/restfully>.
        -C, --engine-config=             Pass the Engine YAML configuration file.
        -u, --user=                      Specify your Grid'5000 username [default=sbadia].
        -q, --queue=                     Specify Grid'5000 OAR queue name (valid queues: admin, testing, besteffort, g5kschool, default)[default=default].
        -g, --gateway=                   Issues SSH commands via the specified gateway.
        -w, --walltime=                  Specify the walltime for the job (in seconds) [default=3600].
        -r, --resources=                 Specify the resources for the job [default=nodes=1].
        -s, --site=                      Site on which to launch the experiment [default=rennes].
        -e, --environment=               Specify the environment to use [default=lenny-x64-base].
        -k, --key=                       Specify the private key to use [default=/home/sbadia/.ssh/id_dsa].
            --deployment-max-attempts=   In case the deployment fails, specifies the number of retries allowed [default=1]
            --deployment-min-threshold=  For a deployment, specifies the minimum percentage (1=100%) of nodes that must have been correctly deployed [default=1]
            --notifications=             A comma-separated list of notification URIs [default=[]]
            --no-deploy                  Attempts to re-use an existing deployment on the same resources.
            --no-submit                  Attempts to re-use an existing job with the same name.
            --no-cleanup                 Do not delete job and deployment at the end of the experiment.
            --no-cancel                  Do not cancel job and deployment even if an error occurs.
            --no-install                 Do no execute the installation phase.
            --no-execute                 Do not execute the execution phase.
            --dev                        Same as --no-cancel --no-cleanup --no-submit --no-deploy. Useful when you need to fine tune a custom engine, or are in development mode.
            --name=                      Name for your experiment [default=Grid5000::Campaign::Engine].
            --log=                       Outputs log messages to the given file. Defaults to STDERR.
        -v, --verbose                    Run in verbose mode.
            --debug                      Run very verbosely (outputs HTTP calls).
            --version                    Display the version.
        -h, --help                       Show this message.

For example, the following command will: ask for 3 nodes in the rennes site with a walltime of 3600 seconds; deploy the lenny-x64-base environment; keep the nodes available at the end of the run (--no-cleanup), so that you can SSH to them easily. Your public key is automatically installed in the authorized keys of the nodes.

  > g5k-campaign -e lenny-x64-base -s rennes -r nodes=3 -w 3600 --no-cleanup -v

See the documentation for Grid5000::Campaign::Engine for the list of options and their semantics.

Usage with custom engines

First, you need to create a subclass of Grid5000::Campaign::Engine, with your own defaults and logic:

  class MyEngine < Grid5000::Campaign::Engine

    set :environment, "lenny-x64-base"
    set :resources, "nodes=4"
    set :walltime, 7200

    on :install! do |env, *args|
      nodes = env[:nodes]
      # configure your nodes
      # e.g. SSH to the first one to install packages and upload some files
      ssh(nodes.first, "root") do |ssh|
        ssh.exec!("apt-get install whatever")
        ssh.sftp.upload!("my_engine/some/file", "/destination/path")
      end
      env
    end

    on :execute! do |env, *args|
      # execute a specific program on your nodes
      env[:nodes].each { |node|
        ssh(node, "root") do |ssh|
          ssh.exec!("./my-program")
        end
      }
      env
    end

  end

Then you can execute it with:

  > g5k-campaign -i path/to/your/my_engine.rb --verbose

See the examples/ directory of the project for advanced examples, and you can find a tutorial at http://grid5000.github.com/tutorials/api/2.0/g5k-campaign-tutorial.html.

See Grid5000::Campaign::Engine#run! source code for an overview of the operations called during the execution of an engine.

How to contribute

Links

Authors