CodingPirates/blockly
1
0
Fork 0
mirror of https://github.com/google/blockly.git synced 2026-01-10 02:17:09 +01:00
Code Issues Packages Projects Releases Wiki Activity

chore(build): Remove build products from the Blockly repository (#6475)

Browse Source 
* feat(build): Make build tasks invoke their prerequisites

  - Divide gulp targets into three kinds: main sequence,
    manually invokable, and script-only.  The first two categories
    automatically invoke their prerequisites.
  - Give (most of) the affected gulp targets shorter and more memorable
    names that could become their npm script names in future.

* feat(build): Make package tasks invoke their prerequisites

  Have the package task invoke the cleanBuildDir (as well as
  cleanPackageDir) and build tasks.  Remove the checkBuildDir
  task as it is now redundant since a fresh build is done every
  time.

* feat(build): Make git tasks invoke their prerequisites

* feat(build): Make cleanup, license [sic] tasks invoke their prerequisites

  Turns out they don't have any, so this commit just classifies
  their gulp targets according to the established scheme.

* feat(build): Make appengine tasks invoke their prerequisites

  In this case prepareDeployDir will eventually depend on package
  but does not for now.

* feat(build): Have npm scripts run npm ci first where applicable

  Have any npm script that have external effects (e.g. publishing an
  npm package, pushing a new version to appengine, or updating GitHub
  Pages) start by running npm ci to ensure that all dependencies are
  up-to-date with respect to package-lock.json.

  (This is done by npm and not a gulp script because gulp itself
  might need updating.  So might npm, but that is less likely to
  make any difference to what gets published/pushed.)

* chore(build): have tests use package target

  Have the tests just run the package target (with debug flags)
  since that runs the the build target automatically.

* feat(tests): Write Closure Compiler output directly to dist/

  Since they are already UMD-wrapped, have Closure Compiler write
  output chunks directly to RELEASE_DIR, i.e. dist/.

* chore(tests): Use freshly-build files in compressed mode.

  Use the freshly-built build/*_compresssed.js files when bootstrapping
  in compressed mode, rather than using the checked-in files in the
  repository root.

  This helps ensure that compressed and uncompressed mode will be
  testing (as closely as possible) the same code.

  Obsoletes #6218 (though the issues discussed there have not actually
  yet been addressed in this branch).

* chore(build): Write intermediate langfiles to build/msg

  Write the results of create_messages.py to build/msg instead of
  build/msg/js.

* fix(build): Use build/msg/en.js instead of msg/messages.js in tests

  This has no direct effect but fixes a long-standing misdesign
  where we are testing against the input to, rather than the output
  of, the language file processing pipeline.

* feat(demos): Use freshly-built files

  Use the freshly-built dist/*_compresssed.js and build/msg/* files
  rather than using the checked-in files in the repository root.

  This helps ensure that these demos are using the most recent
  version of Blockly (even in the develop branch).

* fix(build): Update appengine deployment to include built files

  Modify the prepareDemos task as follows:

  - Use the git index instead of HEAD, so that most local changes
    will be applied (without copying whatever .gitignored cruft
    might be in the local directory).
  - Run clean and build and then copy build/msg and
    dist/*_compressed.js* to the deploy directory.

  This fixes the problem created by the previous commit, wherein the
  demos relied on built files that were not being deployed to
  appengine.

* fix(build): Update GitHub Pages deployment to include built files

  Modify the updateGithubPages task to run clean and build and
  then git add build/msg dist/*_compressed.js*, so that they will
  be included in the deployed pages.

  This fixes the problem created by the previous^2 commit,
  wherein the demos relied on built files that were not being
  deployed to GitHub Pages.

* chore(build): Remove build products from repository

  Remove *_compressed.js* and msg/js/* from the blockly repository.
  Also remove the now-obsolete checkinBuilt gulp task.

* chore(build): Apply relevant changes to test_tasks.js

  Apply changes made to run_all_tests.sh and check_metadata.sh to
  the corresponding parts of their JS replacements in
  test_tasks.js.

* chore(build): Make updates suggested in PR #6475

  - Remove `clean:builddir` and `clean:releasedir` - `clean`
    is sufficient.
  - Remove duplicate `require` from `appengine_tasks.js`.

* feat(build): Use shorter npm script names

  Since scripts that run build tasks now automatically run their
  prerequisite tasks, the previous naming scheme of task `build`
  running all the `build:subtask`s no longe really makes very
  much sense.

  Additionally, following a chat discussion, there seems to be a
  rough consensus to use "messages" to refer to the .json input
  files, and "langfiles" to the generated .js output files.

  Consequently, simplify npm script names by renaming as follows:

  - "generate:langfiles" -> "messages"
  - "build:langfiles" -> "langfiles"
  - "build:js" -> "tsc"
  - "build:deps" -> "deps"
  - "build:compiled" -> "minify"
  - "build:compressed": delete this synonym for "build:compiled",

  ("minify" was chosen as agnostic to Closure Compiler vs. WebPack.)

* chores(build): Add deprecation notice for old scripts

  To reduce potential confusion/frustration, restore the previous
  npm scripts but have them display a deprecation notice instead
  (note that npm prints the script contents before running it, so
  echo is not needed).

* docs(build): Add comments distinguishing 'messages' from 'langfiles'
This commit is contained in:
Christopher Allen
2022-11-03 13:15:10 +00:00
committed by GitHub
parent f95af3614c
commit 52a0d525d7
167 changed files with 334 additions and 56050 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
scripts/gulpfiles/appengine_tasks.js
View File

@@ -14,9 +14,10 @@ var fs = require('fs');
var rimraf = require('rimraf');
var path = require('path');
var execSync = require('child_process').execSync;
const buildTasks = require('./build_tasks.js');
const packageTasks = require('./package_tasks.js');
var packageJson = require('../../package.json');
const demoTmpDir = '../_deploy';
const demoStaticTmpDir = '../_deploy/static';
@@ -33,13 +34,25 @@ function prepareDeployDir(done) {
}
/**
* Copies all files into static deploy directory except for those under
* appengine.
* Copies all files from current git index into static deploy
* directory. We do this rather than just copying the working tree,
* because the working tree is probably full of leftover editor
* backup-save files, vesigial empty directories, etc.
*/
function copyStaticSrc(done) {
execSync(`git archive HEAD | tar -x -C ${demoStaticTmpDir}`,
execSync(`GIT_WORK_TREE='${demoStaticTmpDir}' git checkout-index --all`,
{ stdio: 'inherit' });
done()
done();
}
/**
* Copies needed built files into the static deploy directory.
*
* Prerequisite: clean, build.
*/
function copyBuilt(done) {
return gulp.src(['build/msg/**/*', 'dist/*_compressed.js*'], {base: '.'})
.pipe(gulp.dest(demoTmpDir));
}
/**
@@ -127,10 +140,21 @@ function deployBetaAndClean(done) {
/**
* Prepares demos.
*
* Prerequisites (invoked): clean, build
*/
const prepareDemos = gulp.series(
prepareDeployDir, copyStaticSrc, copyAppengineSrc, copyPlaygroundDeps);
prepareDeployDir,
gulp.parallel(
gulp.series(
copyStaticSrc,
copyAppengineSrc),
gulp.series(
gulp.parallel(buildTasks.cleanBuildDir,
packageTasks.cleanReleaseDir),
buildTasks.build,
copyBuilt),
copyPlaygroundDeps));
/**
* Deploys demos.
@@ -143,7 +167,8 @@ const deployDemos = gulp.series(prepareDemos, deployAndClean);
const deployDemosBeta = gulp.series(prepareDemos, deployBetaAndClean);
module.exports = {
// Main sequence targets. Each should invoke any immediate prerequisite(s).
deployDemos: deployDemos,
deployDemosBeta: deployDemosBeta,
prepareDemos: prepareDemos
}
};

85
scripts/gulpfiles/build_tasks.js
View File

@@ -25,7 +25,7 @@ var closureDeps = require('google-closure-deps');
var argv = require('yargs').argv;
var rimraf = require('rimraf');
var {BUILD_DIR, DEPS_FILE, TEST_DEPS_FILE, TSC_OUTPUT_DIR, TYPINGS_BUILD_DIR} = require('./config');
var {BUILD_DIR, DEPS_FILE, RELEASE_DIR, TEST_DEPS_FILE, TSC_OUTPUT_DIR, TYPINGS_BUILD_DIR} = require('./config');
var {getPackageJson} = require('./helper_tasks');
var {posixPath} = require('../helpers');
@@ -304,15 +304,12 @@ function prepare(done) {
done();
return;
}
return buildJavaScriptAndDeps(done);
return exports.deps(done);
}
const buildJavaScriptAndDeps = gulp.series(buildJavaScript, buildDeps);
/**
* Builds Blockly as a JS program, by running tsc on all the files in
* the core directory. This must be run before buildDeps or
* buildCompiled.
* the core directory.
*/
function buildJavaScript(done) {
execSync(
@@ -327,6 +324,8 @@ function buildJavaScript(done) {
*
* Also updates TEST_DEPS_FILE (deps.mocha.js), used by the mocha test
* suite.
*
* Prerequisite: buildJavaScript.
*/
function buildDeps(done) {
const roots = [
@@ -400,7 +399,7 @@ function buildDeps(done) {
* This task regenrates msg/json/en.js and msg/json/qqq.js from
* msg/messages.js.
*/
function generateLangfiles(done) {
function generateMessages(done) {
// Run js_to_json.py
const jsToJsonCmd = `python3 scripts/i18n/js_to_json.py \
--input_file ${path.join('msg', 'messages.js')} \
@@ -415,10 +414,10 @@ Regenerated several flies in msg/json/. Now run
and check that operation has not overwritten any modifications made to
hints, etc. by the TranslateWiki volunteers. If it has, backport
their changes to msg/messages.js and re-run 'npm run generate:langfiles'.
their changes to msg/messages.js and re-run 'npm run messages'.
Once you are satisfied that any new hints have been backported you may
go ahead and commit the changes, but note that the generate script
go ahead and commit the changes, but note that the messages script
will have removed the translator credits - be careful not to commit
this removal!
`);
@@ -432,7 +431,7 @@ this removal!
*/
function buildLangfiles(done) {
// Create output directory.
const outputDir = path.join(BUILD_DIR, 'msg', 'js');
const outputDir = path.join(BUILD_DIR, 'msg');
fs.mkdirSync(outputDir, {recursive: true});
// Run create_messages.py.
@@ -654,6 +653,8 @@ function compile(options) {
* blockly_compressed.js, blocks_compressed.js, etc.
*
* The deps.js file must be up-to-date.
*
* Prerequisite: buildDeps.
*/
function buildCompiled() {
// Get chunking.
@@ -682,12 +683,14 @@ function buildCompiled() {
.pipe(compile(options))
.pipe(gulp.rename({suffix: COMPILED_SUFFIX}))
.pipe(gulp.sourcemaps.write('.'))
.pipe(gulp.dest(BUILD_DIR));
.pipe(gulp.dest(RELEASE_DIR));
}
/**
* This task builds Blockly core, blocks and generators together and uses
* closure compiler's ADVANCED_COMPILATION mode.
*
* Prerequisite: buildDeps.
*/
function buildAdvancedCompilationTest() {
const srcs = [
@@ -716,35 +719,6 @@ function buildAdvancedCompilationTest() {
.pipe(gulp.dest('./tests/compile/'));
}
/**
* This task builds all of Blockly:
* blockly_compressed.js
* blocks_compressed.js
* javascript_compressed.js
* python_compressed.js
* php_compressed.js
* lua_compressed.js
* dart_compressed.js
* msg/json/*.js
* test/deps*.js
*/
const build = gulp.parallel(
gulp.series(buildJavaScript, buildDeps, buildCompiled),
buildLangfiles,
);
/**
* This task copies built files from BUILD_DIR back to the repository
* so they can be committed to git.
*/
function checkinBuilt() {
return gulp.src([
`${BUILD_DIR}/*_compressed.js`,
`${BUILD_DIR}/*_compressed.js.map`,
`${BUILD_DIR}/msg/js/*.js`,
], {base: BUILD_DIR}).pipe(gulp.dest('.'));
}
/**
* This task cleans the build directory (by deleting it).
*/
@@ -768,17 +742,20 @@ function format() {
.pipe(gulp.dest('.'));
}
module.exports = {
prepare: prepare,
build: build,
javaScriptAndDeps: buildJavaScriptAndDeps,
javaScript: buildJavaScript,
deps: buildDeps,
generateLangfiles: generateLangfiles,
langfiles: buildLangfiles,
compiled: buildCompiled,
format: format,
checkinBuilt: checkinBuilt,
cleanBuildDir: cleanBuildDir,
advancedCompilationTest: buildAdvancedCompilationTest,
}
// Main sequence targets. Each should invoke any immediate prerequisite(s).
exports.cleanBuildDir = cleanBuildDir;
exports.langfiles = buildLangfiles; // Build build/msg/*.js from msg/json/*.
exports.tsc = buildJavaScript;
exports.deps = gulp.series(exports.tsc, buildDeps);
exports.minify = gulp.series(exports.deps, buildCompiled);
exports.build = gulp.parallel(exports.minify, exports.langfiles);
// Manually-invokable targets, with prequisites where required.
exports.prepare = prepare;
exports.format = format;
exports.messages = generateMessages; // Generate msg/json/en.json et al.
exports.buildAdvancedCompilationTest =
gulp.series(exports.deps, buildAdvancedCompilationTest);
// Targets intended only for invocation by scripts; may omit prerequisites.
exports.onlyBuildAdvancedCompilationTest = buildAdvancedCompilationTest;

1
scripts/gulpfiles/cleanup_tasks.js
View File

@@ -81,5 +81,6 @@ function sortRequires() {
};
module.exports = {
// Manually-invokable targets, with prequisites where required.
sortRequires: sortRequires
};

34
scripts/gulpfiles/git_tasks.js
View File

@@ -12,6 +12,7 @@ var gulp = require('gulp');
var execSync = require('child_process').execSync;
var buildTasks = require('./build_tasks');
const packageTasks = require('./package_tasks');
const upstream_url = "https://github.com/google/blockly.git";
@@ -94,7 +95,11 @@ function pushRebuildBranch(done) {
done();
}
// Update github pages with what is currently in develop.
/**
* Update github pages with what is currently in develop.
*
* Prerequisites (invoked): clean, build.
*/
const updateGithubPages = gulp.series(
function(done) {
execSync('git stash save -m "Stash for sync"', { stdio: 'inherit' });
@@ -104,20 +109,27 @@ const updateGithubPages = gulp.series(
done();
},
buildTasks.cleanBuildDir,
packageTasks.cleanReleaseDir,
buildTasks.build,
buildTasks.checkinBuilt,
function(done) {
execSync('git commit -am "Rebuild"', { stdio: 'inherit' });
execSync('git push ' + upstream_url + ' gh-pages --force', { stdio: 'inherit' });
execSync('git add build/msg/* dist/*_compressed.js*', {stdio: 'inherit'});
execSync('git commit -am "Rebuild"', {stdio: 'inherit'});
execSync('git push ' + upstream_url + ' gh-pages --force',
{stdio: 'inherit'});
done();
}
);
module.exports = {
syncDevelop: syncDevelop,
syncMaster: syncMaster,
createRC: createRC,
updateGithubPages: updateGithubPages,
createRebuildBranch: createRebuildBranch,
pushRebuildBranch: pushRebuildBranch
}
// Main sequence targets. Each should invoke any immediate prerequisite(s).
updateGithubPages,
// Manually-invokable targets that invoke prerequisites.
createRC,
// Legacy script-only targets, to be deleted.
syncDevelop,
syncMaster,
createRebuildBranch,
pushRebuildBranch,
};

1
scripts/gulpfiles/license_tasks.js
View File

@@ -17,5 +17,6 @@ function checkLicenses() {
};
module.exports = {
// Manually-invokable targets, with prequisites where required.
checkLicenses: checkLicenses
};

61
scripts/gulpfiles/package_tasks.js
View File

@@ -19,6 +19,7 @@ gulp.replace = require('gulp-replace');
var path = require('path');
var fs = require('fs');
var rimraf = require('rimraf');
var build = require('./build_tasks');
var {getPackageJson} = require('./helper_tasks');
var {BUILD_DIR, RELEASE_DIR, TYPINGS_BUILD_DIR} = require('./config');
@@ -53,41 +54,6 @@ function packageCommonJS(namespace, dependencies) {
});
};
// Sanity check that the BUILD_DIR directory exists, and that certain
// files are in it.
function checkBuildDir(done) {
// Check that directory exists.
if (!fs.existsSync(BUILD_DIR)) {
done(new Error(`The ${BUILD_DIR} directory does not exist. ` +
'Has packageTasks.build been run?'));
return;
}
// Check files built by buildTasks.build exist in BUILD_DIR.
for (const fileName of [
'blockly_compressed.js', // buildTasks.buildCompressed
'blocks_compressed.js', // buildTasks.buildBlocks
'javascript_compressed.js', // buildTasks.buildGenerators
'msg/js/en.js', // buildTaks.buildLangfiles
]) {
if (!fs.existsSync(`${BUILD_DIR}/${fileName}`)) {
done(new Error(
`Your ${BUILD_DIR} directory does not contain ${fileName}. ` +
'Has packageTasks.build been run? Try "npm run build".'));
return;
}
}
done();
}
/**
* This task copies the compressed files and their source maps into
* the release directory.
*/
function packageCompressed() {
return gulp.src('*_compressed.js?(.map)', {cwd: BUILD_DIR})
.pipe(gulp.dest(RELEASE_DIR));
};
/**
* This task wraps scripts/package/blockly.js into a UMD module.
* @example import 'blockly/blockly';
@@ -306,12 +272,12 @@ function packagePHP() {
};
/**
* This task wraps each of the ${BUILD_DIR}/msg/js/* files into a UMD module.
* This task wraps each of the files in ${BUILD_DIR/msg/ into a UMD module.
* @example import * as En from 'blockly/msg/en';
*/
function packageLocales() {
// Remove references to goog.provide and goog.require.
return gulp.src(`${BUILD_DIR}/msg/js/*.js`)
return gulp.src(`${BUILD_DIR}/msg/*.js`)
.pipe(gulp.replace(/goog\.[^\n]+/g, ''))
.pipe(packageUMD('Blockly.Msg', [], 'umd-msg.template'))
.pipe(gulp.dest(`${RELEASE_DIR}/msg`));
@@ -325,10 +291,10 @@ function packageLocales() {
*/
function packageUMDBundle() {
var srcs = [
`${BUILD_DIR}/blockly_compressed.js`,
`${BUILD_DIR}/msg/js/en.js`,
`${BUILD_DIR}/blocks_compressed.js`,
`${BUILD_DIR}/javascript_compressed.js`,
`${RELEASE_DIR}/blockly_compressed.js`,
`${RELEASE_DIR}/msg/en.js`,
`${RELEASE_DIR}/blocks_compressed.js`,
`${RELEASE_DIR}/javascript_compressed.js`,
];
return gulp.src(srcs)
.pipe(gulp.concat('blockly.min.js'))
@@ -399,13 +365,16 @@ function cleanReleaseDir(done) {
/**
* This task prepares the files to be included in the NPM by copying
* them into the release directory.
*
* Prerequisite: build.
*/
const package = gulp.series(
checkBuildDir,
cleanReleaseDir,
gulp.parallel(
build.cleanBuildDir,
cleanReleaseDir),
build.build,
gulp.parallel(
packageIndex,
packageCompressed,
packageBrowser,
packageNode,
packageCore,
@@ -417,15 +386,15 @@ const package = gulp.series(
packageLua,
packageDart,
packagePHP,
packageLocales,
packageMedia,
packageUMDBundle,
gulp.series(packageLocales, packageUMDBundle),
packageJSON,
packageReadme,
packageDTS)
);
module.exports = {
// Main sequence targets. Each should invoke any immediate prerequisite(s).
cleanReleaseDir: cleanReleaseDir,
package: package,
};

29
scripts/gulpfiles/release_tasks.js
View File

@@ -13,7 +13,6 @@ var fs = require('fs');
var gulp = require('gulp');
var readlineSync = require('readline-sync');
var buildTasks = require('./build_tasks');
var gitTasks = require('./git_tasks');
var packageTasks = require('./package_tasks');
var {getPackageJson} = require('./helper_tasks');
@@ -147,27 +146,18 @@ function updateBetaVersion(done) {
done();
}
// Build Blockly and prepare to check in the resulting built files.
const rebuildAll = gulp.series(
buildTasks.cleanBuildDir,
buildTasks.build,
buildTasks.checkinBuilt,
);
// Package and publish to npm.
// Rebuild, package and publish to npm.
const publish = gulp.series(
rebuildAll,
packageTasks.package,
packageTasks.package, // Does clean + build.
checkBranch,
checkReleaseDir,
loginAndPublish
);
// Publish a beta version of Blockly.
// Rebuild, package and publish a beta version of Blockly.
const publishBeta = gulp.series(
updateBetaVersion,
rebuildAll,
packageTasks.package,
packageTasks.package, // Does clean + build.
checkBranch,
checkReleaseDir,
loginAndPublishBeta
@@ -179,12 +169,15 @@ const recompileDevelop = gulp.series(
gitTasks.syncDevelop(),
gitTasks.createRebuildBranch,
updateVersionPrompt,
rebuildAll,
packageTasks.package, // Does clean + build.
gitTasks.pushRebuildBranch
);
module.exports = {
// Main sequence targets. Each should invoke any immediate prerequisite(s).
publishBeta,
publish,
// Legacy target, to be deleted.
recompile: recompileDevelop,
publishBeta: publishBeta,
publish: publish
}
};

30
scripts/gulpfiles/test_tasks.js
View File

@@ -16,7 +16,7 @@ const path = require('path');
const {execSync} = require('child_process');
const rimraf = require('rimraf');
const {BUILD_DIR} = require('./config');
const {BUILD_DIR, RELEASE_DIR} = require('./config');
const runMochaTestsInBrowser =
require('../../tests/mocha/run_mocha_tests_in_browser.js');
@@ -87,12 +87,13 @@ function eslint() {
}
/**
* Run the full usual build process, checking to ensure there are no
* closure compiler warnings / errors.
* Run the full usual build and package process, checking to ensure
* there are no closure compiler warnings / errors.
* @return {Promise} asynchronous result
*/
function buildDebug() {
return runTestCommand('build-debug', 'npm run build-debug');
function build() {
return runTestCommand('build + package',
'npm run package -- --verbose --debug');
}
/**
@@ -110,11 +111,11 @@ function renamings() {
*/
function gzipFile(file) {
return new Promise((resolve) => {
const name = path.posix.join('build', file);
const name = path.posix.join(RELEASE_DIR, file);
const stream = gulp.src(name)
.pipe(gzip())
.pipe(gulp.dest('build'));
.pipe(gulp.dest(RELEASE_DIR));
stream.on('end', () => {
resolve();
@@ -129,7 +130,7 @@ function gzipFile(file) {
* @return {number} 0: success / 1: failed
*/
function compareSize(file, expected) {
const name = path.posix.join(BUILD_DIR, file);
const name = path.posix.join(RELEASE_DIR, file);
const compare = Math.floor(expected * 1.1);
const stat = fs.statSync(name);
const size = stat.size;
@@ -287,14 +288,6 @@ function generators() {
});
}
/**
* Run the package build process, as Node tests depend on it.
* @return {Promise} asynchronous result
*/
function package() {
return runTestCommand('package', 'npm run package');
}
/**
* Run Node tests.
* @return {Promise} asynchronous result
@@ -308,7 +301,7 @@ function node() {
* @return {Promise} asynchronous result
*/
function advancedCompile() {
return runTestCommand('advanced_compile', 'npm run test:compile:advanced');
return runTestCommand('advanced_compile', 'npm run only:compile:advanced');
}
/**
@@ -330,12 +323,11 @@ function reportTestResult() {
// Indivisual tasks.
const testTasks = [
eslint,
buildDebug,
build,
renamings,
metadata,
mocha,
generators,
package,
node,
advancedCompile,
reportTestResult,