require-from-string

Posted on Edited on In Waline:

In recent days, I encountered a requirement, that is, when the project starts, I need to dynamically generate some code and introduce the generated code. At the beginning, my approach was to write the generated code into a file, then require these files, and then succeed. Then delete the file. After doing it, I found out that these files are not necessary to create, can they be directly imported from memory, which reduces the number of files io twice, and require is actually reading the file into memory and then parsing it, so from There is also a theoretical possibility of direct introduction in memory.

Here is a record of the specific approach and the principle behind the analysis.

require-from-string

In fact, this function was developed a long time ago. You just need to download the package “require-from-string” from the npm repository, and then you can directly pass the code string to the function.

One thing to note here is that if you introduce other modules through require-from-string in the code, the relative paths of these modules will be relative to where you call require-from-string.

Here is a simple official example:

1
2
3
4
var requireFromString = require('require-from-string');
 
requireFromString('module.exports = 1');
//=> 1

API

requireFromString(code,

code

Required
Type: string

Module code.

filename

Type: string
Default: ''

Optional filename.

options

Type: object

appendPaths

Type: Array

List of paths, that will be appended to module paths. Useful, when you want to be able require modules from these paths.

prependPaths

Type: Array

Same as appendPaths, but paths will be prepended.

Principle

This package is very simple to use. After using it, let’s take a look at his源码。

The source code is also very simple, with only one index.js file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
'use strict';

var Module = require('module');
var path = require('path');

module.exports = function requireFromString(code, filename, opts) {
	if (typeof filename = 'object') {
		opts = filename;
		filename = undefined;
	}

	opts = opts || {};
	filename = filename || '';

	opts.appendPaths = opts.appendPaths || [];
	opts.prependPaths = opts.prependPaths || [];

	if (typeof code ! 'string') {
		throw new Error('code must be a string, not ' + typeof code);
	}

	var paths = Module._nodeModulePaths(path.dirname(filename));

	var parent = module.parent;
	var m = new Module(filename, parent);
	m.filename = filename;
	m.paths = [].concat(opts.prependPaths).concat(paths).concat(opts.appendPaths);
	m._compile(code, filename);

	var exports = m.exports;
	parent && parent.children && parent.children.splice(parent.children.indexOf(m), 1);

	return exports;
};

The most important thing in this code is the reference to the Nodejs module core module.

Several variables and methods of the Module module are used here.

properties/methodsdesc
module.childrenare the modules required by this module
module.exportsis what the module exposes to the referencer
module.filenamefilename
Module.loadedload is completed, such as in the previous circular reference example, when the load is not completed
module.parentThe first module that requires this module
module.require (id)is the real body of require. I haven’t used it much. It seems that it can be required after the load is completed, but only exports can be seen in other modules, so the module itself needs to be exported.

1
2
3
4
5
6
7
8
9
10
11
12
13
function Module(id, parent) {
  this.id = id;
  this.exports = {};
  this.parent = parent;
  if (parent && parent.children) {
    parent.children.push(this);
  }

  this.filename = null;
  this.loaded = false;
  this.children = [];
}
module.exports = Module;

module._compile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
// Run the file contents in the correct scope or sandbox. Expose
// the correct helper variables (require, module, exports) to
// the file.
// Returns exception, if any.
Module.prototype._compile = function(content, filename) {
  ...(omit here)...
  // create wrapper function
  var wrapper = Module.wrap(content);

  var compiledWrapper = vm.runInThisContext(wrapper, {
    filename: filename,
    lineOffset: 0,
    displayErrors: true
  });

  if (process._debugWaitConnect && process._eval  null) {
    if (!resolvedArgv) {
      // we enter the repl if we're not given a filename argument.
      if (process.argv[1]) {
        resolvedArgv = Module._resolveFilename(process.argv[1], null);
      } else {
        resolvedArgv = 'repl';
      }
    }

    // Set breakpoint on module start
    if (filename = resolvedArgv) {
      delete process._debugWaitConnect;
      const Debug = vm.runInDebugContext('Debug');
      Debug.setBreakPoint(compiledWrapper, 0, 0);
    }
  }
  var dirname = path.dirname(filename);
  var require = internalModule.makeRequireFunction.call(this);
  var args = [this.exports, require, this, filename, dirname];
  var depth = internalModule.requireDepth;
  if (depth = 0) stat.cache = new Map();
  var result = compiledWrapper.apply(this.exports, args);
  if (depth = 0) stat.cache = null;
  return result;
};

Required principle

If the understanding of the above code is not enough, you can take a look at the source code of require. In fact, require-from-string is to intercept the processing of js files in the require source code, and omit some details of the results, eliminating the need for file type judgment, module cache, paths generation, etc.

The specific principle analysis can be seen.这里。

To sum up, it is

  • require actually calls the _load method of the module. This method will first generate the absolute path of the file according to the imported parameter, and then use this absolute path as a key to check the cache (in fact, it is an object). If found, return directly, if not found Continue to compile.

  • compile will first find whether it is the core module, if so, return the require result of the core module, if not, continue.

  • Next is the step in require-from-string, new Module (filename, parent).

  • Next is the assignment of some variables, such as isMain. If the node command runs this file, it is the entry file, and process.mainModule is the current module.

  • The next step is to save this module to the cache.

  • Call tryModuleLoad (module, filename), this method will call module.load, if the call fails, the current module will be removed from the cache.

  • So what does module.load do? The Module._nodeModulePaths (path.dirname (filename)) method generates the path, then gets the extension according to the path, and then calls different methods for different file types, json uses JSON.parse, js calls the module._compile method

  • This _compile parses the module

  • The end is to call NativeModule.wrapper to put the packaged module into the closure

    1
    2
    3
    4
    5
    6
    7
    8
    NativeModule.wrap = function(script) {
        return NativeModule.wrapper[0] + script + NativeModule.wrapper[1];
      };

      NativeModule.wrapper = [
        '(function (exports, require, module, __filename, __dirname) { ',
        '\n});'
      ];