Fixed Error minifying addon JS when external service down

[mattrogowski]

Affected version

2.2.6

Might not be classed as a bug but is an issue one way or another.

XF\Service\AddOn\JsMinifier makes a cURL request to https://closure-compiler.appspot.com/compile, which is currently throwing a 503 error and saying Over Quota This application is temporarily over its serving quota. Please try again later.

This means it's currently impossible to build addons that have minified JS. xf-addon:build-release fails with Unexpected error while minifying JS: Empty result provided by the compiler. It'd probably be preferable to just allow a build without minification, or at least flag that it’s failed because this service is down, otherwise devs need to dig through the core code to figure out why it’s broken. Maybe it could just build without minification and output a warning on the CLI. Removing minification from the build.json gets around the issue, but not an obvious solution unless you dig through the code, and not a long term fix if this service is down for a while/goes away completely.

 

[Xon]

It would be great is XenForo fetched the downloadable version to avoid rate-limiting or the unexpected network dependency.

This isn't particularly hard as the closure compiler is hosted on maven. A tiny bit of add-on code can implement this, but it would be great if this was core;

class JsMinifier extends XFCP_JsMinifier
{
    public function minify()
    {
        $xfRoot = \XF::getSourceDirectory();
        // download from https://mvnrepository.com/artifact/com.google.javascript/closure-compiler
        $closureJar = $xfRoot . DIRECTORY_SEPARATOR . 'closure-compiler-v20210808.jar';
        if (!\file_exists($closureJar))
        {
            $downloadUrl = 'https://repo1.maven.org/maven2/com/google/javascript/closure-compiler/v20210808/closure-compiler-v20210808.jar';
            $response = $this->app->http()->reader()->getUntrusted($downloadUrl);
            if (!$response || $response->getStatusCode() !== 200)
            {
                throw new \ErrorException('Failed to fetch closure-compiler');
            }

            \file_put_contents($closureJar, $response->getBody()->getContents());
        }

        if (\file_exists($closureJar))
        {
            passthru("java -jar {$closureJar} --js {$this->jsPath} --rewrite_polyfills=false --warning_level=QUIET --js_output_file {$this->minPath}", $returnVar);
            if ($returnVar !== 0)
            {
                throw new \ErrorException('Unable to minify ' . $this->jsPath);
            }

            return true;
        }

        return parent::minify();
    }
}

 

[Chris D]

It had been on my mind for a while to allow this to be slightly more robust as we knew the Compiler Service API has its limitations.

The approach is slightly different to above. We've introduced a new $config['development']['closureCompilerPath'] value to src/config.php:

$config['development']['closureCompilerPath'] = \XF::getRootDirectory() . '/development/closure-compiler.jar';

...and if this is set then we use that rather than the Compiler Service API.

YMMV depending on operating system and how you have java configured on your system - it assumes that Java JDK is symlinked to java - but tested okay with macOS and CentOS in fairly typical configurations at least.

Marking this as fixed though no changes have been made to the existing Compiler Service API path as it stands. We just recommend using a locally hosted Closure Compiler JAR starting with XF 2.2.7 if you work on one or more particularly large add-ons and/or need to build releases frequently.

 

[Xon]

I'ld love if it could automatically fetch the closure-compiler jar since discoverability of this sort of feature can be quite low.

I don't have a "good" solution for cross-platform development wise to execute the jar file, but as long as it gives a sane error message or falls back to the webservice it should be OK.

 

[Chris D]

It will be documented in the dev docs but I don't feel that automatically fetching it is the right approach. FWIW if you'd rather continue doing that, this change won't affect your existing class extension so that should continue to work for you as now.

 

[mattrogowski]

Thanks for covering this off - a pretty rare edge case issue, but wasn't ideal having it break when trying to build a release at half past midnight

 

[robdog]

Is this going to be in the next release or have the dev docs been updated? Just ran into this issue myself.

edit NM, I reread Chris' post and see that it is coming out 2.2.7. Thanks for your hard work!

 

[Chris D]

Next release.

 

[Rasmus Vind]

Last night I got this error. I figured it had to do with the Facebook servers being down. But I am still getting this error after Facebook recovered. I haven't changed my JS script since last time it worked. I hope the change you are eluding to also gives more context. Context such as "is the service down or is the compiler flawed because it returns an empty result?" and tips on how to host your own compiler. I am unable to release an update to my site right now because the minifier in the build script fails every time.
I had no idea that JS was minified using an external service. That's... disconcerting to me. I understand the technical difficulties of running a .jar file however. I just hoped there was a default on-premise solution instead.

Try opening:

https://closure-compiler.appspot.com/compile

It says 503 OVER QUOTA

 

[Chris D]

It doesn’t say over quota for me because I haven’t used it in a while

The rate limits are undocumented but there are limits for how many times you can run per hour, per day etc.

Disconcerting, how? It’s a Google app hosted by Google’s app engine.

Closure Compiler Service API Reference | Google for Developers

developers.google.com

If you continually run into limits then we recommend downloading it yourself and setting it as described in this thread above.

 

[Rasmus Vind]

It's up again as of right now.

I think it's disconcerting that my source code is sent through an external service without me knowing. I know it's a minifier, I know the browser runs the code outputted from the minifier. I just like control and to know when I give up control. I am also one of those people who try to limit my interactions with Google.

I had not noticed that XF was updated and you had already addressed this. I see the code now and that I can inject my own jar file. Good job. I will probably do that. I build my own XenForo Docker image. It's just adding a line to get the .jar file and alter my config and we are go on every platform.

 

[Hoffi]

Down again... found this thread today. Lets see how easy the compiler can be installed.

 

[mattrogowski]

Yeah down for me again too. Not sure how the rate limiting on it works but isn't IP based as I haven't built an addon for over a week until now.

 

[TickTackk]

I think it's disconcerting that my source code is sent through an external service without me knowing.

It doesn't take much effort to unminify the contents of minified JS.

 

[Kirby]

XF 2.3 might be the right time to find a replacement for the webservice:

Getting Started with the API | Closure Compiler | Google Developers

developers.google.com