We built a similar proxy at https://wrapapi.com/proxy to allow users to record and replay network requests when turning webpages into an API. I think it's possible to have an end-to-end secure and never-written-to-disk pipeline, provided you trust the provider is doing what they're saying.

The secure pipeline we ended up using involves:

1. Having a HTTPS endpoint for the proxy

2. Forwarding the captures to you immediately via listening WebSocket instances

Note that because the server upon receiving a request can immediately push it to WebSockets instead of relying on polling, no storage is needed

WrapAPI looks neat. Thanks for posting the link, I couldn't find any similar services when we started building debugProxy, although I assumed some must exist.

Whoops, our bad here! Certain accounts had the feature flag turned off randomly -- we should have fixed this, and you can use it now

Yes, definitely for HTTPS in general -- we have detailed instructions for each of 6 different platforms/browsers (e.g., Android, Chrome, Firefox, Windows/Mac desktop). However, the certificate pinning is actually really tough to get around for apps in Android and iOS these days, so those your mileage may vary there

Thanks for linking to this, chipperyman573! The parent comment used to be valid, but we added a dollar amount for the "Business" plan based on this and a few other comments; we are still willing to discuss discounts for non-profits, smaller companies, etc.

Thanks! We used this awesome library called React-Joyride (https://github.com/gilbarbara/react-joyride) which made setting up the product tour a breeze. Since our product tour is on a single-page app, it works quite well.

The most helpful part is that you can pass a callback which will trigger before/during/after each step, which can let you ensure that the state of the page matches what you're expecting. In our case, we use it to make sure that you're switched to the right tab, etc. Take a look! I highly recommend it.

@OhSoHumble: this actually came to mind for us too in terms of building better experiences on top of common government services. I'd love to chat and see how I can help, so shoot me an email at [email protected]

Thanks webninja! The POSTing part is one of the biggest things we were trying to get right while not making it any harder to use than Kimono. Is there anything that was confusing when trying to learn it for the first time? If so, we're still trying to make it easier =)

I think it's pretty close to perfect. I made a sample WrapAPI for http://etfdailynews.com/etf/{{symbol}}/ where the symbol is a 3-4 character ETF ticker symbol like VYM. It's one of the only websites I've found that provides the full breakdown of an ETF's contents. Initially I wasn't sure where to provide sample input such as a handful of ticker 3 character strings since /{{symbol}}/ isn't a GET or a POST value. So under the JSON and Table column I had multiple symbols separated by commas and it took me a little while before I realized that you were only supposed to supply one test value and where I was supposed to supply that test value. But it "clicked" shortly after.

This is designed for at least semi-technical people, but it's really not that hard to give it a try for simple sites. Try watching the video, and shoot me an email at [email protected] if you run into any issues!

We're quite inspired by Kimono, and aim to be just as easy to use while handling use cases beyond scraping (e.g., fully automated form-filling, POST requests, etc.). One of the big feature requests we've been getting has been RSS feeds though, so we're definitely trying to get to full feature parity!

Let us know what page you're trying at [email protected], and I'll take a look and fix it ASAP.