https://dailypypy.org/tags/cloudflare-r2/ Recent content in Cloudflare R2 on 每日拍拍 Hugo -- gohugo.io zh-tw © 2026 每日拍拍 Fri, 28 Aug 2026 10:47:00 +0800 155076163427069952 154825760438254592 https://dailypypy.org/learn/duckdb-remote-parquet-s3/ Fri, 28 Aug 2026 10:47:00 +0800 https://dailypypy.org/learn/duckdb-remote-parquet-s3/ <!--- 1440x768 prompt: masterpiece, best quality, highres, clean anime illustration, japanese anime style, soft shading, flat color design, 1girl, black hair, green eyes, white off-shoulder shirt, black short skirt, from above, full body, crouching beside a small blank translucent data cube, one hand pointing toward layered floating storage shapes, bright curious smile, looking up at viewer, pastel periwinkle blue background, subtle floating translucent file blocks and cloud arcs without symbols or text, neat composition, detailed eyes, cute and smart vibe, minimal background, polished illustration, no text negative prompt: worst quality, bad eye, bad hand, extra limbs, manga, multiple views, monochrome, text, signature dedup note: This remote-object-storage article is a focused follow-up to ../python-duckdb/, which teaches local CSV/Parquet querying and basic SQL. It differs from ../python-fsspec/ by using DuckDB httpfs and typed Secrets rather than a general filesystem abstraction, and from ../python-pyarrow/ by focusing on remote range reads, object-store globbing, Hive partition pruning, projection/filter pushdown, and request diagnostics rather than Arrow schemas and dataset exchange. The Streamlit/Textual DuckDB posts build dashboards; this post stays at the storage and query-engine boundary. ---> <p> <figure> <img class="my-0 rounded-md" loading="lazy" alt="featured" src="./featured.png" /> </figure> </p> <h2 class="relative group">一. 前言：資料在雲端，不代表要先下載 <div id="一-前言資料在雲端不代表要先下載" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"> <a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#%e4%b8%80-%e5%89%8d%e8%a8%80%e8%b3%87%e6%96%99%e5%9c%a8%e9%9b%b2%e7%ab%af%e4%b8%8d%e4%bb%a3%e8%a1%a8%e8%a6%81%e5%85%88%e4%b8%8b%e8%bc%89" aria-label="定位點">#</a> </span> </h2> <p>很多資料管線都有同一個儀式：先把 S3 檔案下載到本機，解壓、查詢，最後再清掉暫存檔。 資料只有幾 MB 時沒什麼感覺；bucket 累積幾十 GB 的 Parquet 後，真正浪費時間的常常不是 SQL，而是搬回根本用不到的欄位與月份。 DuckDB 的 <code>httpfs</code> extension 可以直接讀取 HTTPS、S3 與 S3-compatible object storage。 配合 Parquet 的 columnar layout、row group statistics 與 HTTP Range request，查詢只需要取得相關片段。 這篇拍拍君會完成一條可重用的遠端查詢流程：</p> <ul> <li>用 HTTPS 公開檔案理解 Range read</li> <li>用 DuckDB Secrets 安全連接 S3 與 Cloudflare R2</li> <li>用 glob 與 Hive partition 掃描多檔資料集</li> <li>用 projection/filter pushdown 減少傳輸</li> <li>用 <code>EXPLAIN ANALYZE</code> 找出遠端 I/O 問題</li> </ul> <p>如果還沒碰過 DuckDB，建議先讀 <a href="../python-duckdb/">DuckDB 基礎篇</a>；想理解 Parquet schema 與 dataset，則可搭配 <a href="../python-pyarrow/">PyArrow 實戰</a>。</p>