Skip to content

Commit 89d9400

Browse files
authored
ci: check if external urls in markdown respond (facebook#3581)
1 parent 20e0d1d commit 89d9400

12 files changed

Lines changed: 1919 additions & 263 deletions

File tree

‎docs/network.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -314,7 +314,7 @@ export default class App extends Component {
314314
</TabItem>
315315
</Tabs>
316316

317-
> By default, iOS will block any request that's not encrypted using [SSL](https://hosting.review/web-hosting-glossary/#12). If you need to fetch from a cleartext URL (one that begins with `http`) you will first need to [add an App Transport Security exception](integration-with-existing-apps.md#test-your-integration). If you know ahead of time what domains you will need access to, it is more secure to add exceptions only for those domains; if the domains are not known until runtime you can [disable ATS completely](publishing-to-app-store.md#1-enable-app-transport-security). Note however that from January 2017, [Apple's App Store review will require reasonable justification for disabling ATS](https://forums.developer.apple.com/thread/48979). See [Apple's documentation](https://developer.apple.com/library/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html#//apple_ref/doc/uid/TP40009251-SW33) for more information.
317+
> By default, iOS 9.0 or later enforce App Transport Secruity (ATS). ATS requires any HTTP connection to use HTTPS. If you need to fetch from a cleartext URL (one that begins with `http`) you will first need to [add an ATS exception](integration-with-existing-apps.md#test-your-integration). If you know ahead of time what domains you will need access to, it is more secure to add exceptions only for those domains; if the domains are not known until runtime you can [disable ATS completely](publishing-to-app-store.md#1-enable-app-transport-security). Note however that from January 2017, [Apple's App Store review will require reasonable justification for disabling ATS](https://forums.developer.apple.com/thread/48979). See [Apple's documentation](https://developer.apple.com/library/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html#//apple_ref/doc/uid/TP40009251-SW33) for more information.
318318
319319
> On Android, as of API Level 28, clear text traffic is also blocked by default. This behaviour can be overridden by setting [`android:usesCleartextTraffic`](https://developer.android.com/guide/topics/manifest/application-element#usesCleartextTraffic) in the app manifest file.
320320

‎docs/profiling.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,6 @@ To mitigate this, you should:
135135
- investigate using `renderToHardwareTextureAndroid` for complex, static content that is being animated/transformed (e.g. the `Navigator` slide/alpha animations)
136136
- make sure that you are **not** using `needsOffscreenAlphaCompositing`, which is disabled by default, as it greatly increases the per-frame load on the GPU in most cases.
137137

138-
If these don't help and you want to dig deeper into what the GPU is actually doing, you can check out [Tracer for OpenGL ES](http://www.androiddocs.com/tools/help/gltracer.html).
139-
140138
### Creating new views on the UI thread
141139

142140
In the second scenario, you'll see something more like this:

‎docs/typescript.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ yarn add --dev @tsconfig/react-native @types/jest @types/react @types/react-test
3939
</Tabs>
4040

4141
:::note
42-
This command adds the latest version of every dependency. The versions may need to be changed to match the existing packages used by your project. You can use a tool like [React Native Upgrade Helper](https://react-native-community.github.io) to see the versions shipped by React Native.
42+
This command adds the latest version of every dependency. The versions may need to be changed to match the existing packages used by your project. You can use a tool like [React Native Upgrade Helper](https://react-native-community.github.io/upgrade-helper/) to see the versions shipped by React Native.
4343
:::
4444

4545
2. Add a TypeScript config file. Create a `tsconfig.json` in the root of your project:
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
MIT License
2+
3+
Copyright (c) 2023 Meta Platforms, Inc. and affiliates.
4+
Copyright (c) 2017 David Clark, https://github.com/davidtheclark/remark-lint-no-dead-urls
5+
6+
Permission is hereby granted, free of charge, to any person obtaining a copy
7+
of this software and associated documentation files (the "Software"), to deal
8+
in the Software without restriction, including without limitation the rights
9+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
10+
copies of the Software, and to permit persons to whom the Software is
11+
furnished to do so, subject to the following conditions:
12+
13+
The above copyright notice and this permission notice shall be included in all
14+
copies or substantial portions of the Software.
15+
16+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
17+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
18+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
19+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
20+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
21+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
22+
SOFTWARE.
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
<h1 align="center">Remark Lint: no broken external links</h1>
2+
3+
<p align="center">Remark linter rule to finds broken external links. This is a fork of <a href="https://github.com/davidtheclark/remark-lint-no-dead-urls">remark-lint-no-dead-urls</a></p>
4+
5+
## Usage
6+
7+
Add to your `.remarkrc.js` as a plugin:
8+
9+
```
10+
module.exports = {
11+
plugins: [
12+
'@react-native-website/remark-lint-no-broken-external-links',
13+
],
14+
};
15+
```
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{
2+
"name": "@react-native-website/remark-lint-no-broken-external-links",
3+
"version": "0.0.1",
4+
"private": true,
5+
"description": "Remark linter rule to check for dead urls",
6+
"main": "src/index.js",
7+
"type": "module",
8+
"keywords": [
9+
"remark",
10+
"react-native",
11+
"lint"
12+
],
13+
"files": [
14+
"src/*"
15+
],
16+
"scripts": {
17+
"prettier": "prettier --write \"{src/**/*.js,tests/**/*.js,*.md}\"",
18+
"test": "yarn node --experimental-vm-modules $(yarn bin jest)"
19+
},
20+
"dependencies": {
21+
"got": "^12.5.3",
22+
"unified-lint-rule": "^2.1.1",
23+
"unist-util-visit": "^4.1.2"
24+
},
25+
"devDependencies": {
26+
"dedent": "^0.7.0",
27+
"jest": "^29.4.3",
28+
"remark": "^12.0.1"
29+
}
30+
}
Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
/**
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
*
4+
* This source code is licensed under the MIT license found in the
5+
* LICENSE file in the root directory of this source tree.
6+
*/
7+
8+
import remark from 'remark';
9+
import dedent from 'dedent';
10+
import {jest} from '@jest/globals';
11+
12+
jest.unstable_mockModule('../lib.js', () => ({
13+
fetch: jest.fn(),
14+
}));
15+
16+
const {fetch} = await import('../lib.js');
17+
const plugin = (await import('../')).default;
18+
19+
const processMarkdown = (md, opts) => {
20+
return remark().use(plugin, opts).process(md);
21+
};
22+
23+
describe('remark-lint-no-dead-urls', () => {
24+
beforeEach(() => fetch.mockReset());
25+
26+
test('works with no URLs', () => {
27+
const lint = processMarkdown(dedent`
28+
# Title
29+
30+
No URLs in here.
31+
`);
32+
33+
return lint.then(vFile => {
34+
expect(fetch).toHaveBeenCalledTimes(0);
35+
expect(vFile.messages.length).toBe(0);
36+
});
37+
});
38+
39+
test('works a good, bad a local link', () => {
40+
fetch.mockReturnValueOnce(200).mockReturnValueOnce(404);
41+
42+
const lint = processMarkdown(
43+
dedent`
44+
# Title
45+
46+
Here is a [good link](https://www.github.com).
47+
48+
Here is a [bad link](https://github.com/unified/oops).
49+
50+
Here is a [local link](http://localhost:3000).
51+
`
52+
);
53+
54+
return lint.then(vFile => {
55+
expect(fetch).toHaveBeenCalledTimes(2);
56+
expect(vFile.messages.length).toBe(1);
57+
expect(vFile.messages[0].reason).toBe(
58+
'Link to https://github.com/unified/oops is broken'
59+
);
60+
});
61+
}, 15000);
62+
63+
test('works with definitions and images', () => {
64+
fetch.mockReturnValueOnce(200).mockReturnValueOnce(404);
65+
66+
const lint = processMarkdown(
67+
dedent`
68+
# Title
69+
70+
Here is a good pig: ![picture of pig](/pig-photos/384).
71+
72+
Download the pig picture [here](/pig-photos/384).
73+
74+
Here is a [bad link]. Here is that [bad link] again.
75+
76+
[bad link]: /oops/broken
77+
`,
78+
{
79+
baseUrl: 'http://my.domain.com',
80+
}
81+
);
82+
83+
return lint.then(vFile => {
84+
expect(fetch).toHaveBeenCalledTimes(2);
85+
expect(vFile.messages.length).toBe(1);
86+
expect(vFile.messages[0].reason).toBe('Link to /oops/broken is broken');
87+
});
88+
});
89+
90+
test('skips URLs with unsupported protocols', () => {
91+
const lint = processMarkdown(dedent`
92+
[Send me an email.](mailto:me@me.com)
93+
[Look at this file.](ftp://path/to/file.txt)
94+
[Special schema.](flopper://a/b/c)
95+
`);
96+
97+
return lint.then(vFile => {
98+
expect(fetch).toHaveBeenCalledTimes(0);
99+
expect(vFile.messages.length).toBe(0);
100+
});
101+
});
102+
103+
test('localhost', () => {
104+
const lint = processMarkdown(
105+
dedent`
106+
- [http://localhost](http://localhost)
107+
- [http://localhost/alex/test](http://localhost/alex/test)
108+
- [http://localhost:3000](http://localhost:3000)
109+
- [http://localhost:3000/alex/test](http://localhost:3000/alex/test)
110+
- [https://localhost](http://localhost)
111+
- [https://localhost/alex/test](http://localhost/alex/test)
112+
- [https://localhost:3000](http://localhost:3000)
113+
- [https://localhost:3000/alex/test](http://localhost:3000/alex/test)
114+
`
115+
);
116+
117+
return lint.then(vFile => {
118+
expect(vFile.messages.length).toBe(0);
119+
});
120+
});
121+
122+
test('local IP 127.0.0.1', () => {
123+
const lint = processMarkdown(
124+
dedent`
125+
- [http://127.0.0.1](http://127.0.0.1)
126+
- [http://127.0.0.1:3000](http://127.0.0.1:3000)
127+
- [http://127.0.0.1/alex/test](http://127.0.0.1)
128+
- [http://127.0.0.1:3000/alex/test](http://127.0.0.1:3000)
129+
- [https://127.0.0.1](http://127.0.0.1)
130+
- [https://127.0.0.1:3000](http://127.0.0.1:3000)
131+
- [https://127.0.0.1/alex/test](http://127.0.0.1)
132+
- [https://127.0.0.1:3000/alex/test](http://127.0.0.1:3000)
133+
`
134+
);
135+
136+
return lint.then(vFile => {
137+
expect(vFile.messages.length).toBe(0);
138+
});
139+
});
140+
141+
test.each([
142+
'[Ignore this](http://www.url-to-ignore.com)',
143+
'[Ignore this](http://www.url-to-ignore.com/somePath)',
144+
'[Ignore this](http://www.url-to-ignore.com/somePath?withQuery=wow)',
145+
'[its complicated](http://url-to-ignore.com/somePath/maybe)',
146+
])('skipUrlPatterns for content: %s', markdownContent => {
147+
const lint = processMarkdown(markdownContent, {
148+
skipUrlPatterns: [/^http:\/\/(.*)url-to-ignore\.com/],
149+
});
150+
151+
return lint.then(vFile => {
152+
expect(vFile.messages.length).toBe(0);
153+
});
154+
});
155+
});
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
/**
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
*
4+
* This source code is licensed under the MIT license found in the
5+
* LICENSE file in the root directory of this source tree.
6+
*/
7+
8+
import assert from 'node:assert';
9+
import {URL} from 'node:url';
10+
import {lintRule} from 'unified-lint-rule';
11+
import {visit} from 'unist-util-visit';
12+
import {fetch} from './lib.js';
13+
14+
// Forked from: https://github.com/davidtheclark/remark-lint-no-dead-urls
15+
16+
const linkCache = new Map();
17+
18+
const HTTP = {
19+
OK: 200,
20+
NOT_FOUND: 404,
21+
TOO_MANY_REQUESTS: 429,
22+
};
23+
24+
const uri = {
25+
isLocalhost: url => /^(https?:\/\/)(localhost|127\.0\.0\.1)(:\d+)?/.test(url),
26+
isExternal: url => /(https?:\/\/)/.test(url),
27+
isPath: url => /^\/.*/.test(url),
28+
};
29+
30+
async function cacheFetch(urlOrPath, method, options) {
31+
if (linkCache.has(urlOrPath)) {
32+
return [urlOrPath, linkCache.get(urlOrPath)];
33+
}
34+
35+
const {baseUrl, ...other} = options;
36+
const url = new URL(urlOrPath, baseUrl).toString();
37+
38+
const code = await fetch(url, method, other);
39+
40+
linkCache.set(urlOrPath, code);
41+
return [urlOrPath, code];
42+
}
43+
44+
async function naiveLinkCheck(urls, options) {
45+
return Promise.allSettled(
46+
urls.map(async url => {
47+
try {
48+
return await cacheFetch(url, 'HEAD', options);
49+
} catch (e) {
50+
try {
51+
// Fallback, some endpoints don't support HEAD requests
52+
return await cacheFetch(url, 'GET', options);
53+
} catch (e) {
54+
if (e.code === 'ERR_GOT_REQUEST_ERROR') {
55+
throw e;
56+
}
57+
const code = e.statusCode ?? e?.response?.statusCode ?? e.code;
58+
linkCache.set(url, code);
59+
return [url, code];
60+
}
61+
}
62+
})
63+
);
64+
}
65+
66+
async function noDeadUrls(ast, file, options = {}) {
67+
const urlToNodes = new Map();
68+
69+
const {skipUrlPatterns, ...clientOptions} = options;
70+
71+
// Grab all possible urls from the markdown
72+
visit(ast, ['link', 'image', 'definition'], node => {
73+
const {url} = node;
74+
if (
75+
!url ||
76+
uri.isLocalhost(url) ||
77+
skipUrlPatterns?.some(skipPattern => new RegExp(skipPattern).test(url))
78+
) {
79+
return;
80+
}
81+
82+
// It only makes sense to consider paths when we know the base url. This is useful for images, or cross
83+
// references. There might be false positives when adding new pages that aren't already live.
84+
const isGoodRelativePath = clientOptions.baseUrl && uri.isPath(url);
85+
const isExternalURL = uri.isExternal(url);
86+
if (!isExternalURL && !isGoodRelativePath) {
87+
return;
88+
}
89+
90+
if (!urlToNodes.has(url)) {
91+
urlToNodes.set(url, []);
92+
}
93+
94+
urlToNodes.get(url).push(node);
95+
});
96+
97+
const results = await naiveLinkCheck([...urlToNodes.keys()], clientOptions);
98+
99+
for (const {value, ...other} of results) {
100+
const [url, statusCode] = value;
101+
const nodes = urlToNodes.get(url) ?? [];
102+
103+
if (statusCode === HTTP.OK) {
104+
continue;
105+
}
106+
107+
for (const node of nodes) {
108+
switch (statusCode) {
109+
case 'ENOTFOUND':
110+
file.message(`Link to ${url} is broken, domain not found`, node);
111+
break;
112+
case HTTP.TOO_MANY_REQUESTS:
113+
file.message(`Link to ${url} is being rate limited`, node);
114+
break;
115+
case HTTP.NOT_FOUND:
116+
default:
117+
file.message(`Link to ${url} is broken`, node);
118+
break;
119+
}
120+
}
121+
}
122+
}
123+
124+
export default lintRule('remark-lint:no-broken-external-links', noDeadUrls);
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
/**
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
*
4+
* This source code is licensed under the MIT license found in the
5+
* LICENSE file in the root directory of this source tree.
6+
*/
7+
8+
import got from 'got';
9+
10+
export async function fetch(url, method, options = {}) {
11+
const {statusCode} = await got(url, {
12+
...options,
13+
method,
14+
methodRewriting: true,
15+
});
16+
return statusCode;
17+
}

0 commit comments

Comments
 (0)