My first real post will be an ugly collection of quick notes about Gradle summit 2017 Youtube videos.
Order is the same than this link.
My advice is to watch every videos with a nice comment after /!\ indicator, because you don’t need to watch all videos in order to improve your Gradle skills :)
Main point: Migrate from Ant to Gradle.
Because Ant is XML based, they used Gradle:
- dynamic task generation
- POM customisation (like Android productFlavors)
- Special dependency handling (configurations inspection)
- Usage of init scripts as a way to use XML like build.gradle
10 millions java code, 1000+ projects, 20+ dependency hierarchy
Next step for gradle team after local incremental compilation. Enabled by –build-cache.
Shared build caches:
- HTTP build cache
- Gradle Enterprise build cache
Between developers: no need to rebuild other people’s changes! Recommended sharing strategy: only CI pushes.
- Java projects with Gradle 4.0 (default in .gradle/caches)
- Caching support in Android plugin 3.0 (default in .android/build-cache/)
/!\ GOOD IDEA of parent plugin!! But long talk as well…
Tools: Gradle, Nexus, SonarQube
Parent plugin (10"):
- eliminate boilerplate from build files
- control over dependency versions
- best practices, additional metadata
/!\ GREAT TALK about dependency management and the big-solution used by Netflix!!
Netflix prefers client libraries, means library instead of raw Rest API. So many Netflix internal libs consumed by maven distribution.
How to manage transitive dependencies? Example: A -> B and B -> C, so A -> C (with depends on: ->)
- Semantic version is insufficient: which version is the most stable/etc?
- Consuming libraries is pretty hard!
Publishers lack organization-wide visibility:
- Who is consuming my library?
- Who is using this API that I would like to change?
One solution is monorepo but works with gates (slow publishing).
- Publisher feedback: resolves who depends on me (Astrid tool), and if it’s breaking them by compiling (and tests?) on new version (Niagara tool)
- Managed source: reduce consumer pain by automatically update consumers with new versions of lib
- Distributed refactoring: on all repo, refactor all API calls and issue pull requests en masse (experimental)
Like a consistent distributed monorepo. And seems like Amazon and Linkedin are taking the same way.
/!\ GREAT step by step guide to optimize Gradle builds.
Monitor build for all team, allowing to see how much you spend waiting your builds.
Example: 100 projects, 1M LOC, 50K Tests, Gradle 2.14.1
Configuration | Single-line change | Clean build 4s | 25s | 55s
Use latest version:
./gradlew wrapper --gradle-version 4.0
Measure with Gradle Profiler: performance scenario testing.
- Startup > 1s. Tip: Use daemon with enough memory.
- Configuration time > 1s. Used every time, even during IntelliJ sync.
- Single line change > 10s
- No op build doing any work at all
Below 10s is probably fine.
If big project, break code into smaller modules!
- API / implementation separation
- Compile avoidance
- Incremental compile See blog post.
Parallelise decoupled tests with maxParallelForks and forkEvery config.
- Extract script plugins (apply from “gradle/script.gradle”)
- Extract binary plugins: use buildSrc for heavy lifting
Example presentation about composite builds feature.
Change binary dependencies to a project dependencies.
- cross-repository refactoring
- smooth migration from monorepo to multi-repo
- faster plugin prototyping
- run a continuous, composite build
Like Netflix presentation.
- no versionning between internal components
- unified versions of external dependencies
- Individual versioning of software components
- Freedom to choose dependency version
See http://shipkit.org to auto publish for each CI validation.
Clean is waste of time because:
- Incremental build: no need to change outputs if inputs didn’t change.
- Incremental task input: no need to change all outputs if only some inputs changed (see IncrementalTaskInputs class for plugin developers).
Compile avoidance (typically compile classpath leakage): cascading recompilation resolved by new api/implementation available with new java-library plugin.
Distinguish compile and runtime classpath allows to care only about ABI changes for compile classpath. Even if version change or jar name changed.
Annotation processors disable incremental compilation except if that’s why we have to use apt and compileOnly keywords.
Variant aware dependency management: no need to generate a jar if we need only to compile, classes directory are fine. Or could be generated later while we’re running our changes.
Some hacks around aar format and custom multidex operations. Not useful for common gradle problems.
- New aapt2 to use new gradle incremental (> gradle 3.4).
- Java annotation processor in progress
- Since 2.3 incremental zip task.
New ArtifactTransform allowing to consume aar -> exploded-aar -> 14 artifacts like:
See example project with all best practices.
Most of content is common with his other talk.
Try org.gradle.caching=true especially during git branch changes.
/!\ GREAT TALK about how to resolve Gradle issues, in a hacky way (debug, etc)!!
Partial build (-a) allow to build only current module (DANGEROUS)!
Compiler avoidance for java:
- java -> java-library plugin
- api/implementation configurations
- declare annotation processors in processor classpath
org.gradle.caching=true is enabling build-cache (Gradle 4.0). See 8"40 config.
Use yourkit.com Java Profiler. See 15"30.
Use logs! with –info
Trying Android gradle plugin early:
- AGP 3.0 enable via flag
- define flavor dimensions for API changes
- Disable aapt2 See 20" for hack for old/new dependency systems support.
See 24"03 for hack in order to avoid AS/command line separation of builds…
See 25"53 to see how debug gradle:
git clone https://github.com/gradle/gradle.git ./gradlew idea gr --no-daemon -Dorg.gradle.debug=true task # Wait debugger
See 27"51 to see how use local jar in order to debug AGP.
See 32"22 for Twitter library build example.
See 38" for advices like:
- isolate annotation processor
- use java module when possible
From Uber. Slides
All necessary knowledge before writing a plugin. Slides
How it was implemented and plugin/script examples.
build-scan is free to use!!
/!\ GOOD video for Gradle plugins development.
- Binary plugin > script plugin for tests, etc.
- Prefer statically typed languages: Java, Kotlin or Groovy with @CompileStatic
- Prefer public API via “compile gradleApi()”
- Read performance guide
See plugins dependencies via “buildEnvironment” task.
Capabilities vs conventions allows to share logic but not constants between plugins (see 17"23). For example BaseAndroidPlugin shared between AndroidApplication and AndroidLibrary.
- Use plugin dev plugin: compile ‘java-gradle-plugin’
- Prefer writing custom tasks types
- modeling DSL-like APIs (23"23)
- closure method (25")
Fix guava issues en masse:
- “Netflix rewrite” to define our refactoring rule
- Google BigQuery to find Java files
- Zeppelin/Spark on Dataproc to run our rule against all sources