Web Spider

We want to write a webcrawler that, given a URL A, finds all URLs reachable from A on the same host and protocol.

Given the followingn pages and associated links

https://spider.navidence.dev/1 -> https://spider.navidence.dev/2
https://spider.navidence.dev/1 -> https://spider.navidence.dev/3
https://spider.navidence.dev/2 -> https://spider.navidence.dev/4
https://spider.navidence.dev/3 -> http://spider.navidence.dev/5
https://spider.navidence.dev/3 -> https://www.somewhere-else.com/6
http://spider.navidence.dev/5 -> https://spider.navidence.dev/7

And an initial URL of: https://spider.navidence.dev/1

We expect to get back a list/set (the order is not important) such as the following:

    [
      "https://spider.navidence.dev/1",
      "https://spider.navidence.dev/2",
      "https://spider.navidence.dev/3",
      "https://spider.navidence.dev/4"
    ]

We are not *given* the above set of links, we are initially given only one URL, and must fetch the URLs contents to discover the links it points to.

Explanation

We get 4 because we are recursing.
We skip 5 because it is on the wrong protocol (http)
We skip 6 because it is on the wrong host (www.somewhere-else.com).
We do not see 7 because we do not search pages with URLs that do not meet our criteria, so we do not scan 5 for links.

Start your crawl at 1.html. You should get 1.html, 2.html, 3.html and 4.html in your return list.