How this site's live blog tab pulls articles from DEV, Hacker News and Lobsters with free APIs, without slowing pages or breaking when a source is down.
The short answerFetch third-party content on the server and cache it, so upstream APIs see a few requests an hour. Give every source its own timeout and fallback, put the headlines in the page's HTML so it works without scripts, load the slower extras in the browser, and link out with noopener and nofollow.
Key takeaways
- Fetch third-party content on the server and cache it, so the upstream API sees a few requests an hour instead of one per visitor.
- Give every source its own timeout and its own fallback, and treat the feed as broken only when every source comes back empty.
- Render the headlines on the server with a short cache so the page works before any script runs, and let the browser add the slower extras afterwards.
- Link out with rel noopener and nofollow, and say plainly whose articles they are.
The blog on this site now has a Live blogs tab, which shows what developers are reading right now. None of it is mine, and none of it needs an account or an API key. Getting it onto a mostly static site without making the site slower or more fragile came down to a handful of decisions.
Three free sources, no keys
DEV has a public articles endpoint that takes a tag and a top parameter, where top=7 means the most-reacted-to articles of the last seven days. Hacker News is searchable through Algolia's official API, and the front_page tag returns what is on the front page at that moment. Lobsters serves its own ranking as JSON if you ask for hottest.json.
Three sources rather than one, because each has a different character: DEV is written posts, Hacker News is the widest net, and Lobsters is smaller and strictly technical. Together they are less likely to all be empty, and less likely to all be about the same thing.
Cache once, serve many
The browser never calls those APIs. It asks one route on this site, which calls them on the server. Each upstream request is cached for fifteen minutes by Next.js, and the route's response is cached at the CDN for the same period with stale-while-revalidate, so a visitor just after expiry still gets the previous feed instantly while a fresh one is fetched.
The effect is that the upstream sites see a handful of requests an hour however many people open the tab. That is the polite way to use a free API, and it is also what keeps the feature fast.
Assume every source will fail
Each source is fetched independently, with its own six-second timeout, and the results are collected with Promise.allSettled so one slow or broken source cannot take the others down. Each response is normalised into one small shape, and every URL is checked to be http or https before it is allowed anywhere near a link.
Only when all three come back empty does the route return an error, and that response is deliberately not cached, so the next visitor retries instead of inheriting the failure.
Server-render the list, load the rest later
The blog page is generated on the server and refreshed every fifteen minutes, so the headlines are in its HTML and the page is useful before any script runs. In the browser, opening the tab sends one request to the same cached route, which refreshes the list and adds the pictures. That lookup is slower, so it is kept off the page render itself.
Only headlines, links and the short descriptions the sources publish through their APIs are shown. Nobody's article text is copied, and every item links out to the original.
Say whose it is
Each headline opens a preview page on this site: the cover image, the title and description, the opening of the article where the source provides it, and prominent links to the original and its discussion. Those links open in a new tab with rel noopener, and carry nofollow because this site did not write the page and cannot vouch for it. The preview is marked noindex and names the original as its canonical address, so it never competes with the author's own page. The tab says in plain words where the articles come from and that none of them are mine, and the list is marked up as a list of links rather than as anything I wrote. A live feed is a convenience for the reader, not a way to borrow someone else's credibility.
More writing
Elsewhere on the site
The ideas here get used rather than just written about — the projects are where the animation and performance work lives, and client work covers the production systems. If you want something built, services explains how that works, or just get in touch.