How to avoid re-compiling generated source code

Generating boilerplate source code with sbt works fine:

sourceGenerators in Compile <+= sourceManaged in Compile map { srcDir => DslBoilerplate.generate(srcDir, Seq( "path/to/a/definition/file" )) }

When I run sbt compile this also compiles the generated source code files, thus producing some class files. I just don't want the generated source code to be re-compiled every time I re-compile the project during development.

So, from the class files I made a jar file and used this instead of the generated source/class files (I deleted those). This worked fine, now having access to the generated code through the jar file. Is there a way though to let sbt do the 4 steps (if needed?) in the initial project build?:

<ol> <li>generate source code files</li> <li>compile those files</li> <li>create a jar from the produced class files</li> <li>delete source and class files</li> </ol>

(In this question they use the sbt.IO.jar method to create a jar but there they already have existing files...)

Or is there another better approach than making a jar to avoid re-compiling generated source code?


Update 1 (see update 2 below)

Thanks, Seth, for your answer! It worked great to avoid generating the source files with each project compilation since the cache now remembers that they have been created. I'll certainly use this feature, thanks!

But this was actually not what I had in mind with my original question. Sorry for not being clear enough. It might be clearer if we think of this as 2 transformations happening:

Input file <strong>---1---></strong> Source file (*.scala) <strong>---2---></strong> Target file (*.class)

where the transformations are

<ol> <li>generation of source code (from some information in an input file) and </li> <li>compilation of the generated source code</li> </ol>

This all works fine when I compile the project with sbt compile.

But then if I "rebuild the project" (in IntelliJ), the generated source code (from the sbt compilation) will compile again, and that's what I want to avoid - but at the same time have access to that code. Is there any other way to avoid compilation than placing this code in a jar and then delete the source and target files?

So I tried to continue along that line of thought wrestling with sbt to make it create a source and target jar - still can't make the target jar. This is what I came up with so far (with help from this):

sourceGenerators in Compile += Def.task[Seq[File]] { val srcDir = (sourceManaged in Compile).value val targetDir = (classDirectory in Compile).value // Picking up inputs for source generation val inputDirs = Seq("examples/src/main/scala/molecule/examples/seattle") // generate source files val srcFiles = DslBoilerplate.generate(srcDir, inputDirs) // prepare data to create jars val srcFilesData = files2TupleRec("", srcDir) val targetFilesData = files2TupleRec("", targetDir) // debug println("### srcDir: " + srcDir) println("### srcFilesData: \n" + srcFilesData.mkString("\n")) println("### targetDir: " + targetDir) println("### targetFilesData: \n" + targetFilesData.mkString("\n")) // Create jar from generated source files - works fine val srcJar = new File("lib/srcFiles.jar/") println("### sourceJar: " + srcJar) sbt.IO.jar(srcFilesData, srcJar, new java.util.jar.Manifest) // Create jar from target files compiled from generated source files // Oops - those haven't been created yet, so this jar becomes empty... :-( // Could we use dependsOn to have the source files compiled first? val targetJar = new File("lib/targetFiles.jar/") println("### targetJar: " + targetJar) sbt.IO.jar(targetFilesData, targetJar, new java.util.jar.Manifest) val cache = FileFunction.cached( streams.value.cacheDirectory / "filesCache", inStyle = FilesInfo.hash, outStyle = FilesInfo.hash ) { in: Set[File] => srcFiles.toSet } cache(srcFiles.toSet).toSeq }.taskValue def files2TupleRec(pathPrefix: String, dir: File): Seq[Tuple2[File, String]] = { sbt.IO.listFiles(dir) flatMap { f => { if (f.isFile && f.name.endsWith(".scala")) Seq((f, s"${pathPrefix}${f.getName}")) else files2TupleRec(s"${pathPrefix}${f.getName}/", f) } } }

Maybe I still don't need to create jars? Maybe they shouldn't be created in the source generation task? I need help...


Update 2

Silly me!!! No wonder I can't make a jar with class files if I filter them with f.name.endsWith(".scala"), dohh

Since my initial question was not that clear, and Seth's answer is addressing an obvious interpretation, I'll accept his answer (after investigating more, I see that I should probably ask another question).


You want to use FileFunction.cached so that the source files aren't regenerated every time.

Here's an example from my own build:

sourceGenerators in Compile += Def.task[Seq[File]] { val src = (sourceManaged in Compile).value val base = baseDirectory.value val s = streams.value val cache = FileFunction.cached(s.cacheDirectory / "lexers", inStyle = FilesInfo.hash, outStyle = FilesInfo.hash) { in: Set[File] => Set(flex(s.log.info(_), base, src, "ImportLexer"), flex(s.log.info(_), base, src, "TokenLexer")) } cache(Set(base / "project" / "flex" / "warning.txt", base / "project" / "flex" / "ImportLexer.flex", base / "project" / "flex" / "TokenLexer.flex")).toSeq }.taskValue

Here the .txt and .flex files are input files to the generator. The actual work of generating the source files is farmed out to my flex method, which returns a java.io.File:

def flex(log: String => Unit, base: File, dir: File, kind: String): File = ...

You should be able to adapt this technique to your build.

FileFunction.cached is described in the API doc and in the sbt FAQ under "How can a task avoid redoing work if the input files are unchanged?" (http://www.scala-sbt.org/0.13/docs/Faq.html). (It would be nice if the material on caching was referenced from http://www.scala-sbt.org/0.13/docs/Howto-Generating-Files.html as well; currently it isn't.)


  • NSControl isEnabled only available in OS X v10.0 through OS X v10.9
  • Pandas plotting two graphs on one scale
  • How to retrieve elements of OWL enumerated datatype expression?
  • @Primary equivalent for autowired Spring JPA repositories
  • general concept-java code and cycle clocks
  • html/css: how to create a hexagonal image-placeholder
  • Type mismatch: cannot convert from ListFragment to Fragment
  • object play not found in scala application
  • replacing while loop with list comprehension
  • Checkpointing In ALS Spark Scala
  • CSS Grid, position absolute an element in a css grid item: IMPOSSIBLE
  • Scala: Function returning an unknown type
  • Changing references to deprecated methods C++
  • Spark (Scala) Writing (and reading) to local file system from driver
  • Who propagate bugfixes across branches (corporate development)?
  • DIV instruction jumping to random location?
  • xcode don't localize specific strings
  • Get specific string
  • Not able to aggregate on nested fields in elasticsearch
  • Assign variable to the value in HTML
  • How to use carriage return with multiple line?
  • Abort upload large uploads after reading headers
  • ilmerge with a PFX file
  • java.lang.NoClassDefFoundError: com.parse.Parse$Configuration$Builder on below Lollipop versions
  • What is Eclipse's Declaration View used for?
  • Counter field in MS Access, how to generate?
  • Incrementing object id automatically JS constructor (static method and variable)
  • DirectX11 ClearRenderTargetViewback with transparent buffer?
  • Shallow update not allowed (git > 1.9)
  • Perl system calls when running as another user using sudo
  • Javascript + PHP Encryption with pidCrypt
  • Websockets service method fails during R startup
  • jquery mobile loadPage not working
  • How to include full .NET prerequisite for Wix Burn installer
  • Free memory of cv::Mat loaded using FileStorage API
  • Why can't I rebase on to an ancestor of source changesets if on a different branch?
  • how does django model after text[] in postgresql [duplicate]
  • Qt: Run a script BEFORE make
  • unknown Exception android
  • Running Map reduces the dimensions of the matrices