Developing an application against a fork of Theia

[thegecko]

During last week's dev meeting there was a discussion around ways to develop against Theia, specifically when a developer needs to take advantage of features which aren't yet published or still under proposal.

This is primarily difficult due to Theia being a monorepo and npm/yarn not being able to install packages from within a repo. Short of publishing newer/modified packages to a different registry, setting up this environment has been a little tricky, so I thought I'd share a pattern for using git submodules to bring in Theia packages as a separate workspace. The project can be seen here:

https://github.com/thegecko/theia-dev

I've had to copy a lot of the package dependencies into the root package.json as AFAICT each package doesn't contain all of it's required dependencies.

Are there any thoughts around this approach? Can we make it simpler (perhaps by bringing all dependencies into each package)? Has anyone refined a better setup? Could something like this be turned into a starter project?

[akosyakov]

My concern is that it does not respect yarn.lock files in theia and extension repos, but creates a new one with different node_modules structure and packages' versions. So If it compiles and runs within such repo, it does not mean that it will within theia or extension repos. If it would not be an issue, it can be simplified by moving dev dependencies from the root to a single dev package and listing it as a workspace.

I wonder whether an approach with a local npm repo can work, e.g. with https://github.com/verdaccio/verdaccio.

[JonasHelming]

@thegecko : This has been inactive for quite a while. Can we close this or convert to a dicussion?

[thegecko]

Perhaps convert?

[kachurun]

I found two theoretical ways to solve this problem, and neither of them is particularly pleasant—especially if you want to modify Theia’s code and immediately see the changes in your project.

First approach

Use yalc.

You will need to write a simple script that iterates over each package and publishes it to yalc. In theory, this could be done with Lerna in a single line directly from the scripts section.

Then go to your project and write another script that runs yalc add --pure for all @theia/* packages. You can either list them manually or obtain the list from the yalc cache (which lives somewhere in the home directory).

Next, you need to add overrides for these packages in your package.json, for example:

"@theia/core": "file:.yalc/@theia/core"

Then run npm install.

Whenever you update the Theia code, this process needs to be repeated (fully or partially).

This was the first thing I tried. Theia compiled successfully but did not start because part of the code ended up duplicated. In addition, Bun started complaining about a broken lockfile. I couldn’t fully resolve these issues and initially assumed that the overrides approach was flawed, but it now seems more likely that the real problem lies in how dependencies are installed in my monorepo.

Second approach

In theory this approach should be more reliable, although in practice I’m not entirely sure.

You can deploy a local npm proxy/cache registry, for example using Verdaccio.

Then go to the Theia repository and publish all packages to your local registry. In order to publish an already existing version, you first need to run:

npm unpublish --force && npm publish

After that you can simply install these packages from your local registry.

With Bun I ran into another issue where it cached packages and refused to download them again, although the global cache can be disabled. Other package managers may behave similarly.

Alternatively, you can publish the packages under versions with a suffix like local.N. However, this would result in hundreds of modified package.json files in your Theia patch branch, which is also not very convenient. And if you go so far you can publish your theia fork under your own namespace just in normal npm so it doesn't make much sense to me.

After updating Theia’s code you would still need to reinstall dependencies or change versions, so this approach is not very convenient for development. However, it might be the only way to actually deploy Theia inside your own application.

Unfortunately, this also did not solve my issue. The built code still does not run because part of the codebase appears to be duplicated. It seems the problem lies in the configuration of my monorepo rather than in the way I inject the packages.

[ndoschek]

Hi @kachurun, I've had success using yarn link for this. I recently added a script to the Theia IDE repo that builds against a local Theia framework checkout using yarn links: eclipse-theia/theia-ide#674

Even if you're not building Theia IDE specifically, the approach and script might serve as a good reference for setting up your own project against a local Theia fork. The key idea is to yarn link all @theia/* packages from your local Theia build into your consuming project, which avoids the duplication problem since yarn resolves them to a single copy.
There's also npm link which should work similarly, though I haven't tried it recently.

HTH

[kachurun]

Oh, thats nice!

However I cannot use Yarn, but I assume there isn’t much difference between yalc + overrides/resolutions and npm/bun/yarn link, since the underlying mechanism is basically the same. In theory, all of these approaches should work.

However, in my case even the approach with a custom npm registry does not work. So the issue likely lies in how the package manager (bun in hoisted mode) links packages in my environment (I also have a fairly complex monorepo), or possibly in the changes I made to Theia itself.

Oh yes, I moved my thiea related package from monorepo and it works alone.

[kachurun]

It’s not the easiest approach, but it’s what I use and it finally works for me.

1. Everything in my project always runs inside Docker. It’s not strictly necessary, but it’s convenient for me.

2. I have a container called registry running Verdaccio, bound to registry.localtest.me or localhost:4873.
   In fact, this registry could be public and deployed somewhere in production. In my case I store its database directly in my Git repository, so any published packages end up there. Every package in the registry except Theia is in .gitignore, so only Theia packages are committed to Git, not the whole npm cache.
   Also, each package is just a folder with a *.tgz archive inside, so it doesn’t create thousands of files. Not perfect, but it works and can be improved later if needed.

3. I have this script in the Theia repo: scripts/publish-local.js:

const cp = require('child_process');
const path = require('path');
const fs = require('fs');

const registry = process.env.REGISTRY_URL || 'http://registry.localtest.me';
const pkgPath = path.join(process.cwd(), 'package.json');
const pkg = JSON.parse(fs.readFileSync(pkgPath, 'utf8'));
const spec = `${pkg.name}@${pkg.version}`;

function exec(cmd, args, opts = {}) {
  return cp.spawnSync(cmd, args, { stdio: 'inherit', ...opts });
}

// Unpublish first (the registry may disallow overwriting). Ignore exit code.
exec('npm', ['unpublish', spec, '--registry', registry, '--force'], { stdio: 'pipe' });

const result = exec('npm', ['publish', '--registry', registry, '--force']);
process.exit(result.status || 0);

And in package.json scripts section:

"publish:local": "lerna exec --no-private -- node ../../scripts/publish-local.js"

So I just run this script to publish all packages in the repo. Of course, you can publish only the ones you changed if you want.

To publish a package to Verdaccio with the same version that already exists in the uplink, you need to unpublish it first:

npm unpublish --registry=http://registry.localtest.me
npm publish --force --registry=http://registry.localtest.me

4. In the application where I develop against my Theia fork, I simply specify the versions published in the local registry (usually just latest), so nothing changes here. I just configure my registry URL in bunfig.toml since I use Bun.

5. If you publish a Theia package with the same version and need to [re]install it, clear the cache and force installation:

rm -rf ~/.bun/install/cache/@theia/ node_modules/.cache/@theia
bun i --force --no-verify

---

The advantage of this setup is that it can easily be shared with other developers or used in CI, as long as you have either:

• a registry deployed somewhere accessible to your team (not just localhost)
• the registry database committed to Git.

[ndoschek]

Glad you got it working! Thanks for the detailed write-up @kachurun!

The Verdaccio approach is a solid solution, I use it in other projects as well. Nice tip about the unpublish-before-publish workaround for same-version overwrites! 👍