Allow ignoring empty directories in task inputs

[lptr]

See e.g. this forum request: https://discuss.gradle.org/t/build-cache-issue-how-to-exclude-empty-directories-from-compile/22820

It would be good to be able to declare that an input property does not care about empty directories but only about files. We already do this automatically for classpath resources.
The use case described in the forum is Java source files which also makes good sense.

[big-guy]

I think it could make sense for this to be the default behavior with maybe a way to opt into "directories are meaningful" if there's a use case for it. I think Copy-like tasks are the only ones that are obviously sensitive to directories (but even that's configurable).

As an example, I think we should consider:

src/main/java/com/example/Foo.java
src/main/java/com/example/internal/Bar.java

and

src/main/java/com/example/Foo.java
src/main/java/com/example/internal/Bar.java
src/main/java/com/example/newpackage (empty directory)

as identical. It could be easy for people to get build cache misses without an obvious cause otherwise.

[SUPERCILEX]

@wolfs I'd like to try and fix this, is that ok? I'd be making it so that @Optional allows for both null values and if applied to an @Input[File(s)/Directory], non-extent files.

[SUPERCILEX]

Oh my B, this is the wrong issue. I meant #2016. Either way, I'd like to fix all the input inconsistencies. 😁

[autonomousapps]

Is this issue also about empty source roots? For example, if you have two builds, and in one of those builds an empty root:

module/src/main/java

there will be a cache miss if the task involved has multiple source roots. For example, KotlinCompile. Example build comparison.

[autonomousapps]

I have seen this issue impact users without their having any awareness of it. It is apparently not uncommon to have empty source directories in a project. Because git will not commit an empty directory, only some users will have an empty dir, leading to relocatability issues that impact the remote build cache.

Here is how this situation typically arises, in my experience:

1. A developer will use the IDE (e.g., Android Studio) to create a new module. The IDE helpfully creates main, test, and androidTest source roots. The latter two will contain template test classes. The developer might delete the template tests but not delete the folders.
2. A developer will take an existing large module and factor out a submodule, with the idea of improving the ability of gradle to build bits of the project in parallel. This often leaves behind -- completely accidentally -- empty packages.

In either case, git clean -fd will remove these empty directories, but nevertheless people might go months missing out on remote cache hits because they don't think to do this.

Proposal: Gradle ignore empty source directories by default. Is there a use-case for considering them? Do empty directories ever carry semantic meaning?

[autonomousapps]

As an addendum, sometimes people do something like the below to detect empty source dirs and fail the build in their presence. Unfortunately, this has the side effect of making every SourceTask care about the state of the build script itself. Any change to the build script will now cause these tasks to be out of date. We should do our best to make such ugly workaround unnecessary.

// Empty source dirs produce cache misses, and are not caught by 
// `git status`.
// Fail if we find any.
subprojects {
  tasks.withType(SourceTask).configureEach { t ->
    t.doFirst {
      t.source.visit { FileVisitDetails d ->
        if (d.file.directory && d.file.listFiles().size() == 0) {
          throw new IllegalStateException("Empty source dir: " + d.file.absolutePath + "\nRun `git clean -fd`")
        }
      }
    }
  }
}```

[runningcode]

Yes, this happens quite often. I would estimate that this is happening in every non-trivial project with more than one developer and a version control system.

This can also happen the following case:
3. Move a class (or several classes) from one package to another package within the same module leaving an empty package.

[isker]

This is happening to us with great frequency and makes using the build cache correctly a challenge. Configuring custom task inputs with FileTree is a trap for all but the most trivial cases.

If there's no fix for this problem forthcoming, can the documentation be updated to reflect this fact? Can FileTree#getFiles be prominently called out as the correct solution?

[Tapchicoma]

As temporary workaround if you are using git, you could add following git hook - .git/hooks/post-checkout:

#!/bin/sh
set -e
find . -type d -empty -not -path "./.git/*" -not -path "*/build/*" -not -path "./.idea/*" -not -path "*/.gradle/*" -delete

[ZacSweers]

FYI that Kotlin kapt in 1.4.0 appears to leave empty source directories: https://youtrack.jetbrains.com/issue/KT-41353

[wolfs]

This has been fixed in 6.8 RC1.