52675

Being clever when copying artifacts with Jenkins and multi-configurations

Question:

Suppose that I have a (fictional) set of projects: FOO and BAR. Both of these projects have some sort of multi-configuration option.

FOO has a matrix on axis X which takes values in { x1, ..., xn } (so there are n builds of FOO). BAR has a matrix on axis Y which takes values in { y1, ..., ym } (so there are m builds of BAR).

However, BAR needs to copy some artifacts from FOO. It turns out that Y is a strictly finer partition than n. For example, X might take the values { WINDOWS, LINUX } and Y might be { WINDOWS_XP, WINDOWS_7, DEBIAN_TESTING, FEDORA } or whatever.

Is it possible to get BAR to do some sort of table lookup to work out what configuration of FOO it needs when it copies artifacts across? I can easily write a shell script to spit out the mapping, but I can't work out how to invoke it when Jenkins is working out what it needs to copy.

At the moment, a hacky solution is to have two axes on FOO, on for X and one for Y, and then filter out combinations that don't make sense. But the resulting combination filter is ridiculous and the matrix is very sparse. Yuck.

A solution that I don't like is to parametrise FOO on Y instead: this would be a huge waste of compile time. And, worse, the generated artefacts are pretty big, so even if you did some sort of caching, you'd still have to keep unnecessary copies floating around.

Answer1:

Can't say I fully understand the intricacies if your matrices, but I think I can help you with your actual question

"I can easily write a shell script to spit out the mapping, but I can't work out how to invoke it when Jenkins is working out what it needs to copy"

The <strong>Archive the artifacts</strong> and <strong>Copy artifacts from another project</strong> post-build actions can take java style wildcards, like module/dist/**/*.zip as well as environment variables/parameters, like ${PARAM} for the list or artifacts. You can use commas , to add more artifacts.

The on-page help for <strong>Copy artifacts from another project</strong> states how to copy artifacts of a specific matrix configuration: To copy from a particular configuration, enter JOBNAME/AXIS=VALUE, this is for the Project Name attribute. That project name attribute can also contain params as ${PARAM}

So, in your <strong>BAR</strong> job, have a <strong>Copy Artifacts</strong> build step, with Project Name being FOO/X=${mymapping}. What this will do is: every time a configuration of BAR is run, it will copy artifacts only from FOO with configuration of X=${mymapping}.

Now you need to set the value of ${mymapping} dynamically every time BAR is run. A simple script like this may do the trick:

[[ ${Y:0:7} == "WINDOWS" ]] && mymapping=WINDOWS || mymapping=LINUX

Finally, you need to use <a href="https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin" rel="nofollow"><strong>EnvInject</strong></a> plugin to make this variable available to the rest of the build steps, including the <strong>Copy Artifacts</strong> step.

So, every time BAR configuration runs, it will look at its own configuration axis Y, and if that axis starts with WINDOWS, it will set the ${mymapping} to WINDOWS, else set it to LINUX. This ${mymapping} is then made available to the rest of the build steps. When Copy Artifacts build step is executed, it will only copy artifacts from FOO where the X axis matches ${mymapping} (i.e. either WINDOWS or LINUX).

<h2>Full Setup</h2> <ul><li>Install <a href="https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin" rel="nofollow"><strong>EnvInject</strong></a> plugin.</li> <li>In <strong>BAR</strong> job configuration, tick Prepare an environment for the run (part of EnvInject plugin).</li> <li>Make sure both checkboxes for keeping existing variables are checked.</li> <li>In Script Content copy your script:<br />[[ ${Y:0:7} == "WINDOWS" ]] && mymapping=WINDOWS || mymapping=LINUX</li> <li>Under <strong>Build</strong> steps, configure <strong>Copy Artifacts</strong> build step.</li> <li>Set <strong>Project name</strong> parameter to FOO/X=${mymapping}</li> <li>Configure the rest as usual.</li> </ul>

Recommend

  • How to mock local variables in java? [duplicate]
  • How do you get the value from a TVML textField?
  • Multiple scatterplots using Core Plot and Swift
  • Purescript types for buildQueryString function
  • plotting spatial points over a raster layer in r
  • Intellij Idea Terminal shortcut not working
  • Einsum optimize fails for basic operation
  • Merge arrays by common column values in julia
  • How read between delimiters in php DOM of a XML file?
  • Cannot convert a char value to money. The char value has incorrect syntax
  • C++ String tokenisation from 3D .obj files
  • Is there a chance to get -splash: work for SWT applications that require -XstartOnFirstThread?
  • Scala split a multi line string by lines that contain all hyphens
  • Extracting individual digits from a float
  • Insert records if not exist SQL Server 2005
  • include dlls in visual studio c++ 2008
  • why calling cd shell command through system() or execvp() from a child process won't work?
  • Installing Perl6 and Panda on Ubuntu 15.10. Problems with bootstrap.pl
  • How can I get the full list of running processes on a Mac from a python app
  • Retrieving specified columns from a list of csv files to create a data data frame in R
  • R Split data.frame using a column that represents and on/off switch
  • Filter strings with regex before casting to numeric
  • Plotting line graph with factors in R
  • Cannot upload to OneDrive using the new SDK
  • C# program and C++ DLL compiled for 32-bit system crash on 64-bit system
  • Spark fat jar to run multiple versions on YARN
  • Spring security and special characters
  • Get object from AWS S3 as a stream
  • Excel - Autoshape get it's name from cell (value)
  • Check if a string to interpolate provides expected placeholders
  • How do you troubleshoot character encoding problems?
  • RestKit - RKRequestDelegate does not exist
  • Traverse Array and Display in markup
  • Matrix multiplication with MKL
  • retrieve vertices with no linked edge in arangodb
  • Hits per day in Google Big Query
  • File not found error Google Drive API
  • Linking SubReports Without LinkChild/LinkMaster
  • Converting MP3 duration time
  • Python/Django TangoWithDjango Models and Databases