Scraped text runs as HTML in a generated report

My web scraper wrote scraped titles and links straight into its HTML report, so a page could slip its own markup or a javascript: link into the file. One shared escape function, plus a rule that only http and https addresses become links, fixed it.

Errors or symptoms
scraped text renders as HTML instead of texta javascript: link in the report is clickable
Affects
Rust · HTML
Checked
Depth
one layer down
Tags
rust · html · security · escaping

What went wrong

My web scraper can save its results as an HTML report to open in a browser. The code that wrote it, src/output/html.rs, put every scraped string into the page with a bare writeln!:

writeln!(file, "  <p class=\"url\">Source: {}</p>", result.url)?;

and for each link:

writeln!(
    file,
    "    <li><a href=\"{}\">{}</a></li>",
    link.url, link.text
)?;

Nothing was escaped. A page whose title held <script>alert(1)</script> put a live script tag into the report, and it would run as soon as the file opened. A link whose address was javascript:alert(1) became a working link.

Escape every scraped string

The XML output already had a small escape function. I moved it into src/output/mod.rs as markup_esc, so both outputs share it, and added the single quote:

pub fn markup_esc(s: &str) -> String {
    s.replace("&", "&amp;")
        .replace("<", "&lt;")
        .replace(">", "&gt;")
        .replace('"', "&quot;")
        .replace('\'', "&#39;")
}

The & has to go first. Escape < first and it becomes &lt;, then the & step turns that into &amp;lt;, and the reader sees &lt; on the page instead of <.

Every scraped field now passes through it before it reaches a writeln!: the title, headings, meta tags, image alt text and the source address. Those five characters are the ones OWASP's Cross Site Scripting Prevention Cheat Sheet lists for text going into HTML.

javascript:alert(1) contains none of those five characters, so it comes out of markup_esc unchanged and still works as a link. The danger is the javascript: scheme itself, not the characters. So a second function decides whether an address may become a link at all:

pub fn safe_url(url: &str) -> Option<&str> {
    let lower = url.to_ascii_lowercase();
    (lower.starts_with("http://") || lower.starts_with("https://")).then_some(url)
}

Links and images go through a small anchor helper. An http or https address becomes a link. Anything else, such as a javascript: or data: address, shows as text with the address in a tooltip:

fn anchor(url: &str, text: &str) -> String {
    match safe_url(url) {
        Some(url) => format!("<a href=\"{}\">{}</a>", esc(url), esc(text)),
        None => format!("<span title=\"{}\">{}</span>", esc(url), esc(text)),
    }
}

Allowing two schemes covers data: too, and anything else a browser might run, which a check for javascript: alone would miss.

Screenshot names get ./ in front

The report links each page's screenshot by its file name. A bare name in an href is read as an address, so a name with a colon in it could pass for a scheme. Writing ./ in front always makes it a relative path:

"  <a href=\"./{0}\"><img class=\"shot\" src=\"./{0}\" width=\"320\" loading=\"lazy\" alt=\"Page screenshot\"></a>",

Check it worked

Two unit tests in src/output/mod.rs pin both functions. The one for addresses tries the tricks a page might use:

assert_eq!(safe_url("javascript:alert(1)"), None);
assert_eq!(safe_url(" javascript:alert(1)"), None);
assert_eq!(safe_url("JaVaScRiPt:alert(1)"), None);
assert_eq!(safe_url("data:text/html, <script>"), None);

To see it by hand, scrape a local page with <script>alert(1)</script> in its title and open the report. The tag shows up as text and nothing runs.

Back to all guides