You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Historically, these transforms were unordered, and up until Groovy 4 it was not possible to order them. There were a set of transforms where order did matter. They were grouped by function - the grails-core framework transforms & the grails-data-mapping transforms. To work around the Groovy limitation, Grails introduced both a GlobalGrailsClassInjectorTransformation (for grails-core) and OrderedGormTransformation (for grails-data-mapping). These AST transformations would then call other transformers in their defined order.
The problem with the Grails implementation is that as each annotated node was processed by the Groovy transform, it would call all of the Grails transformers. This mean that the order of processing would often be: Class Node (apply all transformers), Method Node (apply all transformers), etc. While the Groovy 4 TransformWithPriority interface processes all nodes under a single transform, and then moves to the next highest priority transformer.
So far, we have worked towards reproducible builds by moving as many transforms out of the custom grails & data-mapping transformers. We've adopted the Groovy TransformWithPriority interface where possible. We've switched our method iterations & various caches to use collections with determined sort orders. We also have changed our xml files & properties files to write consistent dates based on the presence of a SOURCE_DATE_EPOCH environment variable that is set in CI based on the last commit date.
GroovyTransformOrder
Because the order of AST Transforms was not guaranteed in Groovy 3, we determined the Grails 7 order by extensive debugging. Once we added various combinations on a class and listed the resulting transforms, we created an order registry in the class GroovyTransformOrder. This class lists the transforms in order from highest priority (first run) to lowest priority (last run).
Future Deprecations
Debugging some of these transforms revealed that some are no longer heavy used and have limited test coverage. In some cases, use was even discouraged (i.e. the mixin transform). This ticket deprecates the Mixin transform since AST transforms should be adopted instead. We will remove in a future Grails Version.
Affect on Grails Application
In addition to the security benefits, having a reproducible build means that Grails Applications will likely be reproducible as well. We also can make better use of Gradle caching & transient recompilation issues that have been seen previously will likely cease. As part of preparing for the ASF release process, we have made almost all of the grails-core build parallel & lazy. Gradle tasks such as the views compilers & GSP compilers are now cache-able and should not always rerun when there are no changes. Grails Applications updating to Grails 7 will likely have to update their build files to be compatible with these Gradle enhancements.
Initial PR work
Initial Pull Requests of the reproducibility work are:
A script is checked in under etc/bin that can be run at the root of the project to run 2 back-to-back builds to find differences and compare the results:
etc/bin/test-reproducible-builds.sh
After running this script, the following artifacts are produced under etc/bin/results:
diff.txt - the jars that have differences
first.txt - the jars found in the first run, including their sha256 hash
second.txt - the jars found in the second run, including their sha256 hash
first - the folder containing the jars found in the first run
second - the folder containing the jars found in the second run
A jar path can be copied from diff.txt and the script etc/bin/extract-build-artifact.sh can be run to pull each jar, & extract it to firstArtifact / secondArtifact. From there, the directories can be compared in IntelliJ to see why the jars are different.
Remaining Issues:
26 of 263 jars are mismatching
13 of these jars are the javadoc jars. Grails uses the Gradle Groovydoc task to produce documentation for groovy & java source code. The generated groovydoc is then injected into the javadoc file in place of any javadoc. This task is currently using groovy 3 because Gradle does not have Groovy 4. We are seeing reordering differences on parent methods (i.e. Object / GroovyObject) in the javadoc.
13 of these jars appear to have ordering issues in their class files. The simplest case is the ApplicationTagLib class under the artifact path grails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar. This class does not appear to have any significant transforms being applied to it, but it does make use of multiple traits.
All of the Grails Data mapping transforms have been split from the OrderedGormTransformation except the Transaction & Tenant Transformations. These transformations appear to call each other, and calling the transaction transformation on all nodes prior to the tenant will fail with an access forbidden to a tenantId field. See BookService data service in the grails-datamapping-core-test for an example failure.
Next Steps
This ticket is to track the remaining work for reproducible builds.
ASF Requirements
To begin releasing under the ASF, the Grails project must adhere to it's release policies. The requirements of Automated Release Signing require a reproducible build.
History of Custom Transformations
Grails makes heavy use of:
Historically, these transforms were unordered, and up until Groovy 4 it was not possible to order them. There were a set of transforms where order did matter. They were grouped by function - the grails-core framework transforms & the grails-data-mapping transforms. To work around the Groovy limitation, Grails introduced both a
GlobalGrailsClassInjectorTransformation(for grails-core) andOrderedGormTransformation(for grails-data-mapping). These AST transformations would then call other transformers in their defined order.The problem with the Grails implementation is that as each annotated node was processed by the Groovy transform, it would call all of the Grails transformers. This mean that the order of processing would often be: Class Node (apply all transformers), Method Node (apply all transformers), etc. While the Groovy 4
TransformWithPriorityinterface processes all nodes under a single transform, and then moves to the next highest priority transformer.So far, we have worked towards reproducible builds by moving as many transforms out of the custom grails & data-mapping transformers. We've adopted the Groovy TransformWithPriority interface where possible. We've switched our method iterations & various caches to use collections with determined sort orders. We also have changed our xml files & properties files to write consistent dates based on the presence of a
SOURCE_DATE_EPOCHenvironment variable that is set in CI based on the last commit date.GroovyTransformOrder
Because the order of AST Transforms was not guaranteed in Groovy 3, we determined the Grails 7 order by extensive debugging. Once we added various combinations on a class and listed the resulting transforms, we created an order registry in the class GroovyTransformOrder. This class lists the transforms in order from highest priority (first run) to lowest priority (last run).
Future Deprecations
Debugging some of these transforms revealed that some are no longer heavy used and have limited test coverage. In some cases, use was even discouraged (i.e. the mixin transform). This ticket deprecates the Mixin transform since AST transforms should be adopted instead. We will remove in a future Grails Version.
Affect on Grails Application
In addition to the security benefits, having a reproducible build means that Grails Applications will likely be reproducible as well. We also can make better use of Gradle caching & transient recompilation issues that have been seen previously will likely cease. As part of preparing for the ASF release process, we have made almost all of the
grails-corebuild parallel & lazy. Gradle tasks such as the views compilers & GSP compilers are now cache-able and should not always rerun when there are no changes. Grails Applications updating to Grails 7 will likely have to update their build files to be compatible with these Gradle enhancements.Initial PR work
Initial Pull Requests of the reproducibility work are:
The initial Pull Requests to refactor the entire Grails Gradle Build include:
Current Differing Builds
Of the 263 jars, we have the remaining 26 differing jars:
Testing Changes
A script is checked in under
etc/binthat can be run at the root of the project to run 2 back-to-back builds to find differences and compare the results:After running this script, the following artifacts are produced under
etc/bin/results:diff.txt- the jars that have differencesfirst.txt- the jars found in the first run, including their sha256 hashsecond.txt- the jars found in the second run, including their sha256 hashfirst- the folder containing the jars found in the first runsecond- the folder containing the jars found in the second runA jar path can be copied from
diff.txtand the scriptetc/bin/extract-build-artifact.shcan be run to pull each jar, & extract it tofirstArtifact/secondArtifact. From there, the directories can be compared in IntelliJ to see why the jars are different.Remaining Issues:
26 of 263 jars are mismatching
13 of these jars are the
javadocjars. Grails uses the Gradle Groovydoc task to produce documentation for groovy & java source code. The generated groovydoc is then injected into the javadoc file in place of any javadoc. This task is currently using groovy 3 because Gradle does not have Groovy 4. We are seeing reordering differences on parent methods (i.e. Object / GroovyObject) in the javadoc.13 of these jars appear to have ordering issues in their class files. The simplest case is the
ApplicationTagLibclass under the artifact pathgrails-gsp/plugin/build/libs/grails-gsp-7.0.0-SNAPSHOT.jar. This class does not appear to have any significant transforms being applied to it, but it does make use of multiple traits.All of the Grails Data mapping transforms have been split from the
OrderedGormTransformationexcept the Transaction & Tenant Transformations. These transformations appear to call each other, and calling the transaction transformation on all nodes prior to the tenant will fail with an access forbidden to atenantIdfield. SeeBookServicedata service in thegrails-datamapping-core-testfor an example failure.Next Steps
This ticket is to track the remaining work for reproducible builds.