http,https: add built-in proxy support in http/https.request and Agent
This patch implements proxy support for HTTP and HTTPS clients and
agents in the `http` and `https` built-ins`. When NODE_USE_ENV_PROXY
is set to 1, the default global agent would parse the
HTTP_PROXY/http_proxy, HTTPS_PROXY/https_proxy, NO_PROXY/no_proxy
settings from the environment variables, and proxy the requests
sent through the built-in http/https client accordingly.
To support this, `http.Agent` and `https.Agent` now accept a few new
options:
- `proxyEnv`: when it's an object, the agent would read and parse
the HTTP_PROXY/http_proxy, HTTPS_PROXY/https_proxy, NO_PROXY/no_proxy
properties from it, and apply them based on the protocol it uses
to send requests. This option allows custom agents to
reuse built-in proxy support by composing options. Global agents
set this to `process.env` when NODE_USE_ENV_PROXY is 1.
- `defaultPort` and `protocol`: these allow setting of the default port
and protocol of the agents. We also need these when configuring
proxy settings and deciding whether a request should be proxied.
Implementation-wise, this adds a `ProxyConfig` internal class to handle
parsing and application of proxy configurations. The configuration
is parsed during agent construction. When requests are made,
the `createConnection()` methods on the agents would check whether
the request should be proxied. If yes, they either connect to the
proxy server (in the case of HTTP reqeusts) or establish a tunnel
(in the case of HTTPS requests) through either a TCP socket (if the
proxy uses HTTP) or a TLS socket (if the proxy uses HTTPS).
When proxying HTTPS requests through a tunnel, the connection listener
is invoked after the tunnel is established. Tunnel establishment uses
the timeout of the request options, if there is one. Otherwise it uses
the timeout of the agent.
If an error is encountered during tunnel establishment, an
ERR_PROXY_TUNNEL would be emitted on the returned socket. If the proxy
server sends a errored status code, the error would contain an
`statusCode` property. If the error is caused by timeout, the error
would contain a `proxyTunnelTimeout` property.
This implementation honors the built-in socket pool and socket limits.
Pooled sockets are still keyed by request endpoints, they are just
connected to the proxy server instead, and the persistence of the
connection can be maintained as long as the proxy server respects
connection/proxy-connection or persist by default (HTTP/1.1)
PR-URL: https://github.com/nodejs/node/pull/58980
Refs: https://github.com/nodejs/node/issues/57872
Refs: https://github.com/nodejs/node/issues/8381
Refs: https://github.com/nodejs/node/issues/15620
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
2025-07-02 01:17:07 +02:00
|
|
|
// This tests that when using a proxy with an agent with maxSockets: 1,
|
|
|
|
|
// subsequent requests are queued when the first request is still alive,
|
|
|
|
|
// and processed after the first request completes, and both are sending
|
|
|
|
|
// the request through the proxy.
|
|
|
|
|
|
|
|
|
|
import * as common from '../common/index.mjs';
|
|
|
|
|
import assert from 'node:assert';
|
|
|
|
|
import { once } from 'events';
|
|
|
|
|
import fixtures from '../common/fixtures.js';
|
|
|
|
|
import { createProxyServer } from '../common/proxy-server.js';
|
|
|
|
|
|
|
|
|
|
if (!common.hasCrypto)
|
|
|
|
|
common.skip('missing crypto');
|
|
|
|
|
|
|
|
|
|
// https must be dynamically imported so that builds without crypto support
|
|
|
|
|
// can skip it.
|
|
|
|
|
const { default: https } = await import('node:https');
|
|
|
|
|
|
|
|
|
|
let resolve;
|
|
|
|
|
const p = new Promise((r) => { resolve = r; });
|
|
|
|
|
|
|
|
|
|
// Start a server that delays responses to test queuing behavior
|
|
|
|
|
const server = https.createServer({
|
|
|
|
|
cert: fixtures.readKey('agent8-cert.pem'),
|
|
|
|
|
key: fixtures.readKey('agent8-key.pem'),
|
|
|
|
|
}, common.mustCall((req, res) => {
|
|
|
|
|
console.log('headers received for', req.url, req.headers);
|
|
|
|
|
if (req.url === '/first') {
|
|
|
|
|
// Simulate a long response for the first request
|
|
|
|
|
p.then(() => {
|
|
|
|
|
console.log('Responding to /first');
|
|
|
|
|
res.writeHead(200, { 'Content-Type': 'text/plain' });
|
|
|
|
|
res.end('Response for /first');
|
2025-12-11 00:55:36 +01:00
|
|
|
}).then(common.mustCall());
|
http,https: add built-in proxy support in http/https.request and Agent
This patch implements proxy support for HTTP and HTTPS clients and
agents in the `http` and `https` built-ins`. When NODE_USE_ENV_PROXY
is set to 1, the default global agent would parse the
HTTP_PROXY/http_proxy, HTTPS_PROXY/https_proxy, NO_PROXY/no_proxy
settings from the environment variables, and proxy the requests
sent through the built-in http/https client accordingly.
To support this, `http.Agent` and `https.Agent` now accept a few new
options:
- `proxyEnv`: when it's an object, the agent would read and parse
the HTTP_PROXY/http_proxy, HTTPS_PROXY/https_proxy, NO_PROXY/no_proxy
properties from it, and apply them based on the protocol it uses
to send requests. This option allows custom agents to
reuse built-in proxy support by composing options. Global agents
set this to `process.env` when NODE_USE_ENV_PROXY is 1.
- `defaultPort` and `protocol`: these allow setting of the default port
and protocol of the agents. We also need these when configuring
proxy settings and deciding whether a request should be proxied.
Implementation-wise, this adds a `ProxyConfig` internal class to handle
parsing and application of proxy configurations. The configuration
is parsed during agent construction. When requests are made,
the `createConnection()` methods on the agents would check whether
the request should be proxied. If yes, they either connect to the
proxy server (in the case of HTTP reqeusts) or establish a tunnel
(in the case of HTTPS requests) through either a TCP socket (if the
proxy uses HTTP) or a TLS socket (if the proxy uses HTTPS).
When proxying HTTPS requests through a tunnel, the connection listener
is invoked after the tunnel is established. Tunnel establishment uses
the timeout of the request options, if there is one. Otherwise it uses
the timeout of the agent.
If an error is encountered during tunnel establishment, an
ERR_PROXY_TUNNEL would be emitted on the returned socket. If the proxy
server sends a errored status code, the error would contain an
`statusCode` property. If the error is caused by timeout, the error
would contain a `proxyTunnelTimeout` property.
This implementation honors the built-in socket pool and socket limits.
Pooled sockets are still keyed by request endpoints, they are just
connected to the proxy server instead, and the persistence of the
connection can be maintained as long as the proxy server respects
connection/proxy-connection or persist by default (HTTP/1.1)
PR-URL: https://github.com/nodejs/node/pull/58980
Refs: https://github.com/nodejs/node/issues/57872
Refs: https://github.com/nodejs/node/issues/8381
Refs: https://github.com/nodejs/node/issues/15620
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
2025-07-02 01:17:07 +02:00
|
|
|
} else if (req.url === '/second') {
|
|
|
|
|
// Respond immediately for the second request
|
|
|
|
|
res.writeHead(200, { 'Content-Type': 'text/plain' });
|
|
|
|
|
res.end('Response for /second');
|
|
|
|
|
} else {
|
|
|
|
|
assert.fail(`Unexpected request to ${req.url}`);
|
|
|
|
|
}
|
|
|
|
|
}, 2));
|
|
|
|
|
server.on('error', common.mustNotCall((err) => { console.error('Server error', err); }));
|
|
|
|
|
server.listen(0);
|
|
|
|
|
await once(server, 'listening');
|
|
|
|
|
|
|
|
|
|
// Start a minimal proxy server
|
|
|
|
|
const { proxy, logs } = createProxyServer();
|
|
|
|
|
proxy.listen(0);
|
|
|
|
|
await once(proxy, 'listening');
|
|
|
|
|
|
|
|
|
|
const serverHost = `localhost:${server.address().port}`;
|
|
|
|
|
const proxyUrl = `http://localhost:${proxy.address().port}`;
|
|
|
|
|
|
|
|
|
|
// Create an agent with maxSockets: 1 and proxy support
|
|
|
|
|
const agent = new https.Agent({
|
|
|
|
|
maxSockets: 1,
|
|
|
|
|
proxyEnv: {
|
|
|
|
|
HTTPS_PROXY: proxyUrl,
|
|
|
|
|
},
|
|
|
|
|
ca: fixtures.readKey('fake-startcom-root-cert.pem'),
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
const requestTimes = [];
|
|
|
|
|
|
|
|
|
|
// Make first request that takes longer
|
|
|
|
|
const firstReq = https.request({
|
|
|
|
|
hostname: 'localhost',
|
|
|
|
|
port: server.address().port,
|
|
|
|
|
path: '/first',
|
|
|
|
|
agent: agent,
|
|
|
|
|
}, common.mustCall((res) => {
|
|
|
|
|
console.log('req1 response received');
|
|
|
|
|
let data = '';
|
|
|
|
|
res.on('data', (chunk) => {
|
|
|
|
|
data += chunk;
|
|
|
|
|
});
|
|
|
|
|
res.on('end', common.mustCall(() => {
|
|
|
|
|
console.log('req1 end');
|
|
|
|
|
requestTimes[0] = { path: '/first', data, endTime: Date.now() };
|
|
|
|
|
assert.strictEqual(data, 'Response for /first');
|
|
|
|
|
}));
|
|
|
|
|
}));
|
|
|
|
|
|
|
|
|
|
firstReq.on('socket', common.mustCall((socket) => {
|
|
|
|
|
console.log('req1 socket acquired');
|
|
|
|
|
// Start second request when first request gets its socket
|
|
|
|
|
// so that it will be queued.
|
|
|
|
|
const secondReq = https.request({
|
|
|
|
|
hostname: 'localhost',
|
|
|
|
|
port: server.address().port,
|
|
|
|
|
path: '/second',
|
|
|
|
|
agent: agent,
|
|
|
|
|
}, common.mustCall((res) => {
|
|
|
|
|
let data = '';
|
|
|
|
|
res.on('data', (chunk) => {
|
|
|
|
|
data += chunk;
|
|
|
|
|
});
|
|
|
|
|
res.on('end', common.mustCall(() => {
|
|
|
|
|
requestTimes[1] = { path: '/second', data, endTime: Date.now() };
|
|
|
|
|
assert.strictEqual(data, 'Response for /second');
|
|
|
|
|
|
|
|
|
|
// The two shares the same proxy connection.
|
|
|
|
|
assert.deepStrictEqual(logs, [{
|
|
|
|
|
method: 'CONNECT',
|
|
|
|
|
url: serverHost,
|
|
|
|
|
headers: { 'proxy-connection': 'keep-alive', 'host': serverHost },
|
|
|
|
|
}]);
|
|
|
|
|
proxy.close();
|
|
|
|
|
server.close();
|
|
|
|
|
}));
|
|
|
|
|
}));
|
|
|
|
|
|
|
|
|
|
secondReq.on('error', common.mustNotCall());
|
|
|
|
|
firstReq.end();
|
|
|
|
|
secondReq.end();
|
|
|
|
|
resolve(); // Tell the server to respond to the first request
|
|
|
|
|
}));
|
|
|
|
|
|
|
|
|
|
firstReq.on('error', common.mustNotCall());
|