Skip to content

Meta-Guide.com

Menu
  • Home
  • About
  • Directory
  • Bibliography
  • Videography
  • Pages
  • Index
  • Random
Menu

100 Best Data Pipeline Videos

Notes:

A data pipeline is a series of processes or steps that are used to extract, transform, and load (ETL) data from one or more sources, and to move it into a target destination, such as a data warehouse, database, or other storage system. A data pipeline typically includes a set of tools, processes, and technologies that are used to automate the extraction, transformation, and loading of data from the source(s) to the target destination.

Data pipelines are used in a variety of different contexts and applications, depending on the specific requirements and goals of the organization or project. Some examples of how data pipelines are used include:

  • Extracting data from multiple sources: Data pipelines are often used to extract data from multiple sources, such as databases, files, or other systems, and to consolidate the data into a single, unified format. This can help organizations to gain a more comprehensive and consistent view of their data, and to better understand and analyze the data.
  • Transforming data: Data pipelines are often used to transform data, by cleaning, filtering, or aggregating the data in order to make it more usable or meaningful. For example, a data pipeline might transform raw log data into a structured format, or it might combine data from different sources into a single, integrated dataset.
  • Loading data into a data warehouse: Data pipelines are often used to load data into a data warehouse, which is a type of database that is designed to store large amounts of data, and to support efficient querying and analysis of the data. By using a data pipeline to load data into a data warehouse, organizations can gain access to powerful data management and analysis capabilities, and can use the data to support business intelligence, analytics, or other applications.
  • Providing real-time data: Data pipelines are often used to provide real-time data, by continuously extracting, transforming, and loading data from the source(s) to the target destination. This can enable organizations to gain access to the most up-to-date data, and to make real-time decisions or take real-time actions based on the data.

Overall, a data pipeline is a series of processes or steps that are used to extract, transform, and load data from one or more sources, and to move it into a target destination. Data pipelines are used in a variety of different contexts and applications, and they can help organizations to gain a more comprehensive and consistent view of their data, to transform and clean the data, to load it into a data warehouse, and to provide real-time data.

Wikipedia:

  • Pipeline (software)
  • XML pipeline

See also:

100 Best Amazon AWS Tutorial Videos | Best Amazon DynamoDB Videos | Best AWS Simple Workflow Videos


[249x Oct 2017]

  • Building modern data pipelines with Spark on Azure HDInsight
  • Streaming Data Pipelines on Apache Mesos: Lessons Learned
  • Setting up an Effective Data Pipeline for LiveOps | James Gwertzman
  • [SIGNAL London] Build a Serverless Data Pipeline
  • LD4P Data Pipeline Sprint 0 Demo
  • Building Robust Streaming Data Pipelines with Apache Spark – Zak Hassan, Red Hat
  • Demytifying the Data Pipeline
  • Webinar S4N: Data Streams & Data Pipelines
  • Intro to Building Data Pipelines in Python with Luigi
  • LD4P Data Pipeline Sprint 0 Demo
  • Data Pipelines
  • 3.3 – 3.1.2 “Big Data Pipelines: The Rise of Real-Time” [Cloud Computing Applications, Part 2: Bi…
  • 4.4 – Typical Analytical Operations in Big Data Pipelines [Big Data Integration and Processing]
  • 4.3 – Aggregation Operations in Big Data Pipelines [Big Data Integration and Processing]
  • 4.2 – Some High-Level Processing Operations in Big Data Pipelines [Big Data Integration and Proce…
  • Erin Shellman Interview – Data Pipelines at Zymergen with Airflow
  • Why a Data Pipeline and Why you need a Data Engineer – Code Mania 101
  • One Data Pipeline to Rule Them All
  • #BDAM: Data Pipelines in Kubernetes, by Sean Suchter, Pepperdata
  • Managing Data Pipelines for Big Data Success
  • Implementing a next generation data pipeline in eMAG
  • Ask RBK: How do I get the right data pipelines?
  • Scott Wiseman – Kafka: Building a Data Pipeline – BSDC 2017
  • ETL and big data Building simpler data pipelines
  • ETL and Big Data Building Simpler Data Pipelines
  • ETL and Big Data Building Simpler Data Pipelines
  • How the Alooma Data Pipeline works with the Snowflake Data Warehouse
  • [Open Academy 2017/I] Pálma Dániel – Data Pipeline építés a Luigi keretrendszer segítségével
  • SummerSOC 2017 – “Big Data Pipelines: Towards A Reference Architecture” E. Syed (Philips LR)
  • 51 . AWS DATA PIPELINE
  • Data pipeline at Spotify – from the inception to the production – Rafal Wojdyla, Spotify
  • Introduction to Text Analytics with R – Part 3: Data Pipeline
  • Riccardo Magliocchetti – Dai dati alla visualizzazione: la mia prima data pipeline
  • Automate Your Data Pipeline – Attunity Compose for Hive
  • #bbuzz 17: Sean Braithwaite – Mechanics of Data Pipelines
  • How to Write Batch or Streaming Data Pipelines with Apache Beam in 15 mins with James Malone
  • Building a Unified Data Pipeline with Apache Spark and XGBoost with Nan Zhu
  • Create, with Intel, an IoT Gateway and Establish a Data Pipeline to AWS IoT
  • Workshop – Data pipelines for your business KPIs and KRAs
  • Data Pipeline Evolution – Ali King – Codemotion Amsterdam 2017
  • Asynchronous Data Pipeline = AWS (S3 & SQS) + FME Cloud – FME UC 2017
  • Use Containerized Camel, Spark and Kafka to Create a Data Pipeline – Zak Hassan (DevNet Create 2017)
  • Best Practices for Building a Cloud Data Pipeline
  • Streaming Data Pipelines with Brooklin–Samarth Shetty, LinkedIn (5/24/17)
  • Building the Data Pipeline Final Project
  • Data Pipeline Project Demo
  • GOTO 2017 • Cloud Native Data Pipelines • Sid Anand
  • [Matúš Cimerman: Building AI data pipelines using PySpark @ PyData Bratislava Meetup #3]
  • Building Robust and Scalable Data Pipelines with Kafka
  • Sam Kitajima Kimbrel One Data Pipeline to Rule Them All PyCon 2017
  • Jason Myers Leveraging Serverless Architecture for Powerful Data Pipelines PyCon 2017
  • Data Pipelines with Firebase and Google Cloud (Google I/O ’17)
  • Aaron Knight Build a data pipeline with Luigi PyCon 2017
  • ctcs 2017 – Learnings from building a marketing data pipeline using Hadoop, Spark, and Airflow
  • Evolving Your Data Pipeline – Yali Sassoon – Snowplow San Francisco Meetup #2
  • Apache Spark as a Platform for Powerful Custom Analytics Data Pipeline: Talk by Mikhail Chernetsov
  • DA332 – Orchestrating Big Data Pipelines with Azure Data Factory (Lace Lofranco)
  • SFBigAnalytics 2017-05-10 GoPro data pipeline & Analytics
  • On the path to building an event-monitoring data pipeline for storage microservices
  • YOW! Nights February 2017 Lynn Langit – Google Cloud Data Pipeline Patterns
  • Katharina Jarmul – Building Data Pipelines with Python
  • Learn CDAP: Preview for Batch Data Pipelines
  • Learn CDAP: Preview for Realtime Data Pipelines
  • Scalable data pipelines with shapeless and cats – Marcus Henry, Jr.
  • Logstash Monitoring: X-Ray Vision for Your Data Pipeline
  • James Brook / Streaming data pipelines with Apache Beam and Google Cloud / Sanoma TechTalks
  • Developing Real-Time Data Pipelines with Apache Kafka
  • Build Simplest Data Pipeline
  • Deploying Fast Data Pipelines
  • Developing Real-Time Data Pipelines with Apache Kafka
  • Orchestrating Big Data Pipelines with Azure Data Factory
  • Orchestrating Big Data Pipelines with Azure Data Factory
  • GO Channels and async Data Pipelines patterns. Lessons learned.
  • Realtime Data Pipelines with Elixir GenStage – Peter Hastie
  • Data Pipelines in core.async w/ Priyatam Mudivarti
  • Australia 2017 Orchestrating Big Data Pipelines with Azure Data Factory
  • Orchestrating Big Data Pipelines with Azure Data Factory
  • Orchestrating Big Data Pipelines with Azure Data Factory
  • Developing Real-Time Data Pipelines with Apache Kafka
  • Data Pipelines with Spark & DataStax Enterprise
  • Google Cloud and Data Pipeline Patterns
  • DataDirect Hybrid Data Pipeline: Planning Your Installation
  • DataDirect Hybrid Data Pipeline: Troubleshooting
  • Evolving Your Data Pipeline – Christophe Bogaert – Snowplow London Meetup #4
  • Developing Real-Time Data Pipelines with Apache Kafka
  • Social Media Social Data and Python: 12 – Building complex data pipelines
  • Build an Agile and Elastic Big Data Pipeline
  • Building Realtime Data Pipelines with Kafka Connect & Spark Streaming by Ewen Cheslack-Postava
  • Data PipeLine Import failure
  • Staging Reactive Data Pipelines using Kafka as the Backbone
  • RubyConf Taiwan 2016 — How to write complex data pipelines in Ruby by Kazuyuki Honda
  • Data Pipeline – New
  • scala.bythebay.io: Modern Software Architectures and Data Pipelines Panel
  • scala.bythebay.io: Moon, Complete big data pipeline with Apache Zeppelin
  • Apache Beam to design your data pipelines by Jean Baptiste Onofre at JBCNConf 2016
  • Continuously Deploying Big Data Pipelines with Amaterasu – Yaniv Rodenski & Eyal Ben Ivri (Eng)
  • BigQuery & Building Data Pipelines–Tips from Full Stack Analytics #JOINData 2016
  • Data Pipeline Evolution – Ali King
  • AWS re:Invent 2016: JustGiving: Serverless Data Pipelines, ETL & Stream Processing (BDM303)
  • Building A Data Pipeline on Google Container Engine at Arbor (by Joshua Kwan)
  • “Data Pipelines for Small, Messy and Tedious Data”, Vladislav Supalov
  • New Data Pipeline Transforms How Clouds Access Data
  • #BDAM: Designing Modern Data Pipelines with Apache Kafka
  • Creating a data pipeline with Couchbase Mobile – Couchbase Connect 2016
  • Building Data Pipelines with Spark and StreamSets (Pat Patterson)
  • Focusing on your data pipelines and forgeting about the rest – Pierre Borckmans
  • DataDirect Hybrid Data Pipeline: Overview
  • Web Tech Topic #13 – Data Pipeline in Paktor & Optus / Unit Test With RSpec
  • DataDirect Hybrid Data Pipeline: Deployment Scenarios
  • Femi Anthony | Creating Python Data Pipelines in the Cloud
  • Hunter Owens | Building Your First Data Pipelines
  • GOTO 2016 • Resilient Predictive Data Pipelines • Siddharth “Sid” Anand
  • Staging Reactive Data Pipelines Using Kafka
  • Building a serverless data pipeline with AWS
  • Realtime Data Pipeline w Spark & Cassandra + Mesos (Rahul Kumar, Sigmoid) | C* Summit 2016
  • Hunter Owens | Luigi & Data Pipelines
  • Zero to Hero Data Pipeline – from MongoDB to Cassandra – Demi Ben Ari @ Panorays (Eng)
  • Challenges & opportunities around elastic data pipelines’ – Jörg Schad
  • Custom Data Pipelines using Kubernetes & Dockers
  • Hydrator: Open Source, Code Free Data Pipelines, by Jon Gray CEO, Cask
  • Big Data Day LA 2016 – Hydrator: Open Source, Code-Free Data Pipelines, Jon Gray, CEO, Cask Data
  • Alooma – The Data Pipeline You Can Trust
  • Data Pipeline and BI Team–Data Modeling of Data Warehouse and BI (Power Pivot and Tableau)
  • data.bythebay.io: Monal Daxini, Netflix Keystone – Streaming Data Pipeline @Scale in the Cloud
  • SF Big Analytics: Building/Runn­i­ng Netflix’s Data Pipeline using Apache Kafka
  • Using Python to Build a GIS Data Pipeline for Rural-Urban Classification – PyConSG 2016
  • Marco Bonzarini – Building data pipelines in python
  • Scalable Streaming Data Pipelines with Redis — Avram Lyon, Scopely
  • The Evolution of Big Data Pipelines at Intuit
  • Building and Managing Large Scale Data Pipelines with Complex Dependencies Using Apache Oozie
  • Let’s build a Service Oriented Data Pipeline
  • Jeff Bowen: Listen To Your Users — Your Data Pipeline
  • #BDAM: Building Data pipelines with Cask Hydrator, by Gokul Gunasekaran from Cask
  • Databricks’ Data Pipelines: Journey And Lessons Learned
  • Building Realtime Data Pipelines with Kafka Connect and Spark Streaming
  • Streaming Data Pipelines With Container
  • Webinar: Building Data Pipelines with SMACK Designing Storage Strategies for Scale
  • AWS Knowledge Center Videos: “How do I create an AWS Data Pipeline role?”
  • Anne Matthies – Zero-Administration Data Pipelines using AWS Simple Workflow
  • Mercedes Coyle – Build Serverless Realtime Data Pipelines with Python and AWS Lambda – PyCon 2016
  • Jakob van Santen – The IceCube data pipeline from the South Pole to publication
  • Data Pipelines Webinar with Priya Joseph and PowerToFly
  • Austin Cassandra Users – Laying down the SMACK on your data pipelines
  • Marco Bonzanini – Building Data Pipelines in Python
  • A Machine Learning Data Pipeline – PyData SG
  • Marco Bonzanini – Building Data Pipelines in Python
  • Ali Zaidi – 10 things I learned about writing data pipelines in Python and Spark.
  • Using Cask Hydrator to easily build reliable and repeatable data pipelines on Hadoop
  • How to Build Data Pipelines for Real Time Applications with SMACK & Apache Kafka
  • Containerized data pipelines with mesos and EMR
  • Developing Real-Time Data Pipelines with Apache Kafka
  • SnapLogic Live: Spark Data Pipelines
  • Scoring and retraining ML models using managed data pipelines Final
  • PHP UK Conference 2016 – Samantha Quiñones – Real Time Data Pipelines
  • Developing Elastic Data Pipelines
  • Short Footage #8 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Short Footage #7 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Short Footage #3 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Short Footage #2 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Short Footage #5 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Short Footage #1 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Short Footage #6 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Short Footage #9 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Short Footage #4 – Big Data, AWS & the Data Pipeline. Distributed MPP & Analytics with HPCC
  • Data pipelines from zero to solid
  • Building Realtime Data Pipelines with Kafka Connect and Spark Streaming
  • Samantha Quiñones – Real-Time Data Pipelines (243)
  • Making your Data Flow With the Data Pipelines Pilot at London’s Calling 2016
  • Lars Albertsson – Data pipelines
  • Architecting on Amazon Web Services: Creating a Data Pipeline
  • How to build Big Data Pipelines for Hadoop using OSS
  • How Vitria builds real-time data pipelines
  • Hadoop 02 (Data Pipeline – Hadoop v1.0)
  • Orchestrating a climate modeling data pipeline (Andre R. Erler)
  • Telligent Data Pipeline – Overview
  • Multi-application data pipelines with Robin
  • O’Reilly Media Webcast: Building Real-Time Data Pipelines
  • Dwolla – Building Scalable Event-Driven Data Pipelines for Payments
  • Migrating Data Pipeline from MongoDB to Cassandra – Demi Ben-Ari @ Windward (Heb)
  • Exploring Real-Time Data Pipelines
  • Microsoft Ignite 2015 Build Hybrid Big Data Pipelines with Azure Data Factory and Azure HDInsight
  • code.talks 2015 – Data Pipeline mit Apache Kafka (Moritz Siuts & Robert von Massow)
  • 20151014 Meetup Data Management – Fabien Janssens – “Data Pipeline” within AXA
  • Dylan Barth, Stuart Coleman: A beginner’s guide to building data pipelines with Luigi
  • Data Pipelines: Big Data Meets Salesforce
  • BDSBTB 2015: Neville Li, Scala Data Pipelines at Spotify
  • SF Scala @Spotify: Neville Li, Macros in Data Pipelines
  • WOLFconnect Data Pipeline
  • Embedding Python Scripts into CloverETL Data Pipeline
  • Intro to Building Data Pipelines in Python with Luigi
  • Yagnik Khanna – Critical pipe fittings: What every data pipeline requires
  • Airflow An open source platform to author and monitor data pipelines
  • Autodesk Building a Self Service Big Data Pipeline
  • Designing data pipelines for autonomous and trusted analytics
  • Building a Data Pipeline with Distributed Systems
  • In-memory data pipeline and warehouse at scale using Spark, Spark SQL, Tachyon & Parquet (2)
  • In-memory data pipeline and warehouse at scale using Spark, Spark SQL, Tachyon & Parquet (1)
  • No coding approach for Data pipelines, Data Discovery & Ad-hoc analysis – on Hadoop & Spark
  • Gwen Shapira – Designing Agile Data Pipelines
  • #bbuzz 2015: Ema Iancuta & Radu Chilom – In-memory data pipeline and warehouse at scale
  • 2015 Track3 4 Big Data Pipelines
  • 2015 Track 3.4 Big Data Pipelines
  • 2013-05 Data Migration with Data Pipeline
  • Reactive data-pipelines with Spring XD and Kafka
  • Build Hybrid Big Data Pipelines with Azure Data Factory and Azure HDInsight
  • New Workflows for Building Data Pipelines
  • SILK: Spark Data Pipeline- Reliable and Accurate Food Dataset- Hesamoddin Salehian (Myfitnesspal)
  • Macros in Data Pipelines
  • The Data Pipeline: Byte4 assignment
  • The Data Pipeline Byte4
  • Scala Data Pipelines for Music Recommendations
  • A Data Pipeline in Talend – 2
  • A Data Pipeline in Talend – 1
  • How to Build a Data Pipeline on Apache Kafka by Etsy Developers
  • Tim Spurway – Disco Distributed Multi Stage Data Pipelines
  • David Pick – Building a Data Pipeline with Clojure and Kafka
  • AWS re:Invent 2014 | (BDT303) Construct ETL Pipeline w/ AWS Data Pipeline, Amazon EMR & Redshift
  • RICON 2014: David Pick, Braintree – Building a Real-Time Data Pipeline with Clojure and Kafka
  • Data Pipeline at Tapad – Toby Matejovsky
  • Building a Unified “Big Data” Pipeline in Apache Spark by Aaron Davidson at ScalaMatsuri2014
  • Insight Data Science – Bitcoin data pipeline
  • AWS Data Pipeline (italiano)
  • Apache Tez: Accelerating Hadoop Data Pipelines
  • Building a Data Pipeline from Scratch – Joe Croback, Project Florida
  • M.A.R.S. Data Pipeline Proof of Concept
  • 0603 Monitoring the Data Pipeline Lessons Learned at Hulu
  • 0604 Building a Unified Data Pipeline in Apache Spark
  • 0603 Building a Hadoop Powered Commerce Data Pipeline
  • 05 – Getting Started with Microsoft Big Data – Operationalize your Big Data Pipeline
  • Big Data Pipelines and Use Cases at StumbleUpon – SF Data Mining Meetup Talk
  • Building Scalable, Flexible Data Pipelines for Big Data, Vivek Ganesan 20140224
  • Apache Kafka: Real-time Streaming and Data Pipelines with Apache Kafka by Joe Stein
  • Amy Unruh: The ‘Internet Of Things’ and Data Pipelines – DevFest Praha 2013
  • Big Data Integration & Analytics Data Flows with AWS Data Pipeline (BDT207) | AWS re:Invent 2013
  • Deploying the ‘League of Legends’ Data Pipeline with Chef (ARC205) | AWS re:Invent 2013
  • Developer Day #7 – Simplifying the Data Pipeline
  • Developer Day #5 – Data Pipeline
  • Designing Data Pipelines Using Hadoop
  • Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
  • Basic Troubleshooting with AWS Data Pipeline
  • How to build Big Data Pipelines for Hadoop using OSS
  • Process Web Logs with AWS Data Pipeline, Amazon EMR, and Hive
  • First Look AWS Data Pipeline
  • AWS re:Invent BDT 201: AWS Data Pipeline: A guided tour
  • GA Boot Camp: Illumina — Working with Data (Pipeline workflow)
  • Google I/O 2012 – Building Data Pipelines at Google Scale
  • Jay Kreps Hadoop Summit 2011 Building Kafka and LinkedIn’s Data Pipeline
  • Bill Graham Hadoop Summit 2011 Using a Hadoop data pipeline to build a graph
  • Google I/O 2010 – Data pipelines with Google App Engine

Popular Content

New Content

Virtual Human Systems: A Generalised Model (2021)

 

Contents of this website may not be reproduced without prior written permission.

Copyright © 2011-2025 Marcus L Endicott

©2025 Meta-Guide.com | Design: Newspaperly WordPress Theme