/
Building Eclipse itself
Building Eclipse itself
- Frits Jalvingh
Owned by Frits Jalvingh
Building Eclipse itself is horror, because it is not at all clear what needs to happen to actually develop and build it. This should, once, become a reasonable description of how to - for instance - add a fix to the compiler. But for now it's just a list of links and snippets.
Building a subdirectory from the Aggregator clone:
cd eclipse.jdt.core/org.eclipse.jdt.core mvn -Pbuild-individual-bundles clean verify
If JUnit tests fail out of the box you can skip them using:
mvn -Pbuild-individual-bundles clean verify -Dmaven.test.skip=true -DskipTests=true
You need to run the same command in the other parts of the eclipse.jdt.core dir, at the very least:
- org.eclipse.jdt.core
- org.eclipse.jdt.compiler.tool
- org.eclipse.jdt.compiler.apt
Compile these IN ORDER because the ecj compiler contains all of these.
To build the batch compiler issue that inside eclipse.jdt.core. After that you need to run:
cd scripts ant -f export-ecj.xml
You need to patch it though (the file is apparently old) so that it uses target/classes instead of bin as the source for binaries for the tool and apt projects.
After this the compiler can be found at /home/jal/git/eclipse/eclipse.platform.releng.aggregator/ecj-export/ecj-head.jar
This build process is a disaster.
Discovering the reason for "Errors occurred while compiling module xxxx" errors in IntelliJ Idea.
It is impossible to use the ecj compiler for Java 9+ (like 4.9.x) with IntelliJ because compilation would often fail with completely useless messages like "Internal compiler error" or "Errors occurred while compiling module xxxx".
While I am a huge fan of Jetbrains IDE I think they really suck like big hairy balls here: the messages are incomprehensible and lack all information needed to solve them. Enabling debug logging on the build process also does not help: it shows nothing of value. Most importantly: it does not show the stdout/stderr of the Eclipse compiler process they spawn- and that is really needed to solve this issue.
After some experimentation (and some help from Jetbrains, see this ticket) the root cause of the issue is that the ECJ compiler does not send all errors through the javax.tools.Compiler API. This seems to be true when you look at the ECJ code base: the startup of the compiler seems littered with throw Exception clauses which do not lead at all to some diagnostic.
To find out what is causing trouble is not trivial. In the end I resorted to adding exception logging statements in (amongst others) EclipseCompilerImpl to log the options passed to the compiler and (more importantly) to log the exceptions that were thrown during compilation:
public boolean call() {
try {
try(PrintWriter pw = new PrintWriter(new FileOutputStream(new File("/tmp/ecj" + System.nanoTime() + ".log")))) {
pw.println("COMPILING: ");
this.options.forEach((k, v) -> pw.println(" " + k + " = " + v));
} catch(Exception x) {
x.printStackTrace();
}
and in the catch clauses at the end of this block:
(PrintWriter pw = new PrintWriter(new FileOutputStream(new File("/tmp/ecj" + System.nanoTime() + ".log")))) {
e.printStackTrace(pw);
} catch(Exception x) {}
The options for the compile (found by patching Main) are:
Starting compile got writers allocated compiler 1 Allocated compiler 2 add option -noExit add option -deprecation add option -encoding add option --release add option -s add string option -noExit add string option -g add string option -deprecation add string option -proceedOnError add string option -encoding add string option UTF-8 add string option --release add string option 6 add string option -s add string option /home/jal/git/domui/java9/common/sass-compiler/target/generated-test-sources/test-annotations add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/AutomaticSassTests.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/Comments.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/VariableGuarded.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/parser/ParserTest.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/css/EmptyBlock.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/AbstractDirectoryScanningSassTests.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/Mixins.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/css/Reindeer.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/resolvers/VaadinResolverTest.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/css/Media.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/ParentImports.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/internal/expression/ArithmeticExpressionEvaluatorTest.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/css/Properties.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/AutomaticSassTestsBroken.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/W3ConformanceTests.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/internal/util/StringUtilTest.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/SassLangTests.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/css/Interpolation.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/Functions.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/SassTestRunner.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/Imports.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/Nesting.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/tree/ImportNodeTest.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/ControlDirectives.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/css/Selectors.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/UrlModes.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/SassLangTestsBroken.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/NestedProperties.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/VaadinThemes.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/AbstractTestBase.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/MixinExtraParameters.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/CompassImports.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/ParentSelector.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/Variables.java add uri file:/home/jal/git/domui/java9/common/sass-compiler/src/test/java/com/vaadin/sass/testcases/scss/Extends.java
2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at java.io.FileInputStream.open0(java.base@10.0.1/Native Method) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at java.io.FileInputStream.open(java.base@10.0.1/FileInputStream.java:220) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at java.io.FileInputStream.<init>(java.base@10.0.1/FileInputStream.java:158) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.util.Util.getFileByteContent(Util.java:399) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.tool.EclipseFileObject.getCharContent(EclipseFileObject.java:145) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.tool.EclipseCompilerImpl$3.getContents(EclipseCompilerImpl.java:258) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.parser.Parser.parse(Parser.java:11909) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.parser.Parser.parse(Parser.java:11881) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.parser.Parser.dietParse(Parser.java:10286) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.Compiler.internalBeginToCompile(Compiler.java:837) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.Compiler.beginToCompile(Compiler.java:391) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:445) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:423) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.batch.Main.performCompilation(Main.java:4699) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.tool.EclipseCompilerImpl.call(EclipseCompilerImpl.java:98) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.eclipse.jdt.internal.compiler.tool.EclipseCompiler$1.call(EclipseCompiler.java:195) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.javac.JavacMain.compile(JavacMain.java:194) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.java.JavaBuilder.compileJava(JavaBuilder.java:448) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.java.JavaBuilder.compile(JavaBuilder.java:318) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.java.JavaBuilder.doBuild(JavaBuilder.java:243) 2018-09-23 17:01:37,805 [ 580486] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.java.JavaBuilder.build(JavaBuilder.java:201) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.IncProjectBuilder.runModuleLevelBuilders(IncProjectBuilder.java:1322) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.IncProjectBuilder.runBuildersForChunk(IncProjectBuilder.java:997) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.IncProjectBuilder.buildTargetsChunk(IncProjectBuilder.java:1070) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.IncProjectBuilder.buildChunkIfAffected(IncProjectBuilder.java:960) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.IncProjectBuilder.buildChunks(IncProjectBuilder.java:792) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.IncProjectBuilder.runBuild(IncProjectBuilder.java:377) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.incremental.IncProjectBuilder.build(IncProjectBuilder.java:185) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.cmdline.BuildRunner.runBuild(BuildRunner.java:138) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.cmdline.BuildSession.runBuild(BuildSession.java:300) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.cmdline.BuildSession.run(BuildSession.java:133) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.cmdline.BuildMain$MyMessageHandler.lambda$channelRead0$0(BuildMain.java:229) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.cmdline.BuildMain$MyMessageHandler$$Lambda$91/1444403508.run(Unknown Source) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.service.impl.SharedThreadPoolImpl.lambda$executeOnPooledThread$0(SharedThreadPoolImpl.java:42) 2018-09-23 17:01:37,806 [ 580487] INFO - j.compiler.server.BuildManager - BUILDER_PROCESS [stdout]: at org.jetbrains.jps.service.impl.SharedThreadPoolImpl$$Lambda$68/1990385139.run(Unknown Source)
Logging the exceptions like above proved that the build actually did not fail on exception. Instead what seems to cause the trouble is this, the end of EclipseCompilerImpl#call():
try(PrintWriter pw = new PrintWriter(new FileOutputStream(new File("/tmp/ecjfinish" + System.nanoTime() + ".log")))) {
pw.println("Finished " + this.globalErrorsCount);
} catch(Exception x) {}
if (this.globalErrorsCount == 0)
return true;
For the two failing modules this returned 20 and 46, respectively.
Adding another output file to Logger inside Main to report errors and warnings finally gives a cause:
/home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Handler.java:4: error: The import org.w3c.dom.Document cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Handler.java:5: error: The import org.w3c.dom.Element cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Handler.java:92: error: Document cannot be resolved to a type /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Handler.java:92: error: Element cannot be resolved to a type /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Handler.java:98: error: Element cannot be resolved to a type /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Handler.java:103: error: Element cannot be resolved to a type /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Handler.java:108: error: Element cannot be resolved to a type /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Matcher.java:3: error: The import org.w3c.dom.Document cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Matcher.java:4: error: The import org.w3c.dom.Element cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Matcher.java:42: error: Document cannot be resolved to a type /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/data/Matcher.java:42: error: Element cannot be resolved to a type /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/LoggerConfigPage.java:4: error: The import org.w3c.dom.Document cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/LoggerConfigPage.java:5: error: The import org.w3c.dom.Element cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/LoggerConfigPage.java:6: error: The import org.w3c.dom.Node cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/LoggerConfigPage.java:7: error: The import org.w3c.dom.NodeList cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/LoggerConfigPage.java:33: error: The import javax.xml.parsers.DocumentBuilder cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/LoggerConfigPage.java:34: error: The import javax.xml.parsers.DocumentBuilderFactory cannot be resolved /home/jal/git/domui/java9/to.etc.domui/src/main/java/to/etc/domui/log/LoggerConfigPage.java:35: error: The import javax.xml.parsers.ParserConfigurationException cannot be re
and more of that ilk.