How to resolve dependency conflict out of your control
My project uses Apache Twill and Cassandra Java Driver. Twill uses Guava 13.0.1 while Cassandra Java Driver uses Guava 16.0.1. Even worse…
My project uses Apache Twill and Cassandra Java Driver. Twill uses Guava 13.0.1 while Cassandra Java Driver uses Guava 16.0.1. Even worse, because Twill and Cassandra Java Driver both leak the usage of Guava, my project also directly uses Guava (both versions).
The problem is that Guava 16.0.x is not backward compatible to 13.0.x and it is non-trivial work to upgrade from 13.0.x to it.
This isn’t that rare when you have some most common dependencies like Guava or Apache HttpComponent. Many open source projects use these libraries (of different versions), you ARE going to hit it if you have more than a few dependencies. Usually, the solution is simply to upgrade. We could just patch Twill’s Guava usage to 16.0.x, but Twill is not a small library and its usage of Guava is so fundamental that it takes a lot of effort to go with this option.
One quick and easy way is to shade the dependency. By shading, it means relocating/renaming the dependency to avoid conflicts.
Both Gradle and Maven have plugins for shading. Okay, I can’t believe we’re still using Maven. Anyway, Maven shade plugin can directly relocate classes in addition to building a flat jar. That is while packaging all of a project’s dependencies into a single jar, it can also do optional class relocation if you want.
Here’s how to use it: add the following to Twill parent project POM’s pluginManagement
and you get a shaded flat jar.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>twill-${project.version}-shaded</finalName>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>shaded.com.google.common</shadedPattern>
</relocation>
</relocations>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
Note that I relocate all Guava classes to a new package name but NO code change is required. The class relocation is at binary level.
To use this custom built, shaded flat jar, we could use a local repository in our project:
<repositories>
<repository>
<id>local-repo</id>
<url>file://${basedir}/lib</url>
</repository>
</repositories>
and install the shaded flat jar into this local repository:
mvn \
org.apache.maven.plugins:maven-install-plugin:2.3.1:install-file \
-Dfile=./twill-0.6.0-incubating-shaded.jar \
-DgroupId=org.apache.twill \
-DartifactId=twill \
-Dversion=0.6.0-incubating-shaded \
-Dpackaging=jar \
-DlocalRepositoryPath=lib
Finally, use the shaded dependency in our project:
<dependency>
<groupId>org.apache.twill</groupId>
<artifactId>twill</artifactId>
<version>0.6.0-incubating-shaded</version>
</dependency>
If you look into the shaded flat jar files, you can see the relocated Guava classes.
# unzip -t lib/twill-0.6.0-incubating-shaded.jar | grep shaded
Archive: lib/twill-0.6.0-incubating-shaded.jar
testing: shaded/ OK
testing: shaded/com/ OK
testing: shaded/com/google/ OK
testing: shaded/com/google/common/ OK
testing: shaded/com/google/common/collect/ OK
.
.
.
Usually, that’s all we need to do. The Twill’s usage of Guava 13.0.1 is now renamed and isolated in that flat jar. But remember I said both Twill and Cassandra Java Driver exposes Guava classes to our code?
Because of this, whenever Guava is used for Twill (like passing Guava-type based objects as arguments), I have to use the package “shaded.com.google.common”. But when Guava is used for Cassandra Java Driver, I still use package “com.google.common”.
This does require code change though, a simple find-n-replace import package name. Not ideal, but it has to be like this.