Search on S3 Using AWS Lambda

Hacker News

Hi everyone, During the last 3 months, I worked on the Quickwit search engine to adapt it for AWS Lambda runtime. I wrote a blog post to announce the beta release [0] and two tutorials to quickstart: - Indexing and searching 20 million logs entries dataset [1]. - Implementing an E2E use case where an application generates data, uploads it in batches to S3. Users can then search through an HTTP API authenticated with an API Key [2]. On the performance side, I observed on a 20 million log dataset that search is sub-second, and analytics queries take from 1 to 4 second. On the cost side, obviously the compute using Lambda is fairly expensive, but the huge win is that it scales to 0. Few other solutions do. For instance OpenSearch Serverless has a fixed cost around $700/months. On the search side, thanks to indexing, queries are usually way cheaper than scanning services like CloudWatch (at least 50x cheaper). I would love to get the feedback from the community and see how I can push this further. [0]: Blog post https://quickwit.io/blog/quickwit-lambda-beta [1]: 20 million log dataset search tutorial https://quickwit.io/docs/get-started/tutorials/tutorial-aws-... [2]: E2E use case tutorial https://quickwit.io/docs/guides/e2e-serverless-aws-lambda

Visit product View on Hacker NewsJan 21, 2024

Share card

Actual performance

6points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: user, using, open · Missing: mac, agents, macos

73%73% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: ide, io · Missing: https docs, excited, just released

64%64% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Indie HackersFits the IH revenue-focused audience · Strong signals: started · Missing: supports, reddit linkedin, podcasting

52%52% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: month, users, way · Missing: mobile apps, ios, personal

44%44% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: users · Missing: plus, platform, intuitive

36%36% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

16%16% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

2%2% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Honey Buckets – Find out who is snooping through your AWS S3 buckets57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Honey Buckets – Find out who is snooping through your AWS S3 buckets

Hacker News2

Container registry on a budget using AWS S338%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Container registry on a budget using AWS S3

Hacker News23

Pare down your S3 Bill; with `du` for AWS S356%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Pare down your S3 Bill; with `du` for AWS S3

Hacker News22

Provision AWS S3 buckets safely with least privilege using Terraform47%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Provision AWS S3 buckets safely with least privilege using Terraform

Hacker News7

AWS Lambda Terraform Cookbook with S3,SQS,DynamoDB,API-GW,Kinesis Integ61%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

AWS Lambda Terraform Cookbook with S3,SQS,DynamoDB,API-GW,Kinesis Integ

Hacker News5

Electron-based AWS S3 uploader app55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Electron-based AWS S3 uploader app

Hacker News4

Jeager Plugin for AWS S3/GCS/Azure Blobs28%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Jeager Plugin for AWS S3/GCS/Azure Blobs

Hacker News2

AWS S3 Manager – Desktop Client for AWS S344%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

AWS S3 Manager – Desktop Client for AWS S3

Hacker News2

SwiftProxy – Access AWS S3 Using OpenStack Swift API55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

SwiftProxy – Access AWS S3 Using OpenStack Swift API

Hacker News8

AWS S3 outage test from across the world50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

AWS S3 outage test from across the world

Hacker News77

Browse all products Analyze your launch