This is a repo with links to everything you'd ever want to learn about data engineering
Upserts, Deletes And Incremental Processing on Big Data.
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
type-class based data cleansing library for Apache Spark SQL
Code for blog at: https://www.startdataengineering.com/post/docker-for-de/
SparkSQL.jl enables Julia programs to work with Apache Spark data using just SQL.
FLaNK AI Weekly covering Apache NiFi, Apache Flink, Apache Kafka, Apache Spark, Apache Iceberg, Apache Ozone, Apache Pulsar, and more...
Delta Lake Examples
Repository for Lab “Distributed Big Data Analytics” (MA-INF 4223), University of Bonn
This repository contains all the projects and labs I worked on while pursuing professional certificate programs, specializations, and bootcamp. [Areas: Deep Learning, Machine Learning, Applied Data Science].
PySpark es una biblioteca de procesamiento de datos distribuidos en Python que permite procesar grandes volúmenes de datos en clústeres utilizando el framework Apache Spark, ofreciendo un alto rendimiento y un conjunto de herramientas integradas para el análisis y manejo de datos a gran escala.
Trigger spark-submit in Golang. A Go implementation of famous SparkLauncher.java.
Connect to SQL Server using Apache Spark
Link Prediction is about predicting the future connections in a graph. In this project, Link Prediction is about predicting whether two authors will be collaborating for their future paper or not given the graph of authors who collaborated for atleast one paper together.
Examples usages for cleanframes library
Enrichment pipeline for CUR / FOCUS reports which adds energy and carbon data allowing to report and reduce the impact of the your cloud usage.
A Capstone Project that covers several aspects of Data Engineering (Data Exploration, Cleaning, Modeling, Pipelining, Processing)
Add a description, image, and links to the apachespark topic page so that developers can more easily learn about it.
To associate your repository with the apachespark topic, visit your repo's landing page and select "manage topics."