Start Data Engineering
DECEMBER 11, 2022
1. Introduction 2. Source & Sink 2.1. Source Replayability 2.2. Source Ordering 2.3. Sink Overwritability 3. Data pipeline patterns 3.1. Extraction patterns 3.1.1. Time ranged 3.1.2. Full Snapshot 3.1.3. Lookback 3.1.4. Streaming 3.2. Behavioral 3.2.1. Idempotent 3.2.2. Self-healing 3.3. Structural 3.3.1. Multi-hop pipelines 3.3.2. Conditional/ Dynamic pipelines 3.3.3.
KDnuggets
DECEMBER 16, 2022
Many aspiring Data Scientists, especially when self-learning, fail to learn the necessary math foundations. These recommendations for learning approaches along with references to valuable resources can help you overcome a personal sense of not being "the math type" or belief that you "always failed in math.".
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Confessions of a Data Guy
DECEMBER 10, 2022
There once was a day when no one used DataFrames that much. Back before Spark had really gone mainstream, Data Scientists were still plinking around with Pandas a lot. My My, what would your mother say? How things have changed. Now everyone wants a piece of the DataFrame pie. I mean it tastes so good, […] The post Dataframe Showdown – Polars vs Spark vs Pandas vs DataFusion.
Christophe Blefari
DECEMBER 16, 2022
Prepping me to deliver Christmas' Data News ( credits ) Hey you, the end of the year is coming soon. I really liked this year with you. It was super fun to write every Friday of the year my opinion on data topics, I don't know yet if next year I'll be able to pull out stuff without repeating myself, I hate repeating myself, but for sure I'll try and I'll continue.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Seattle Data Guy
DECEMBER 10, 2022
If you haven’t started looking for ways to improve your data analytics budget for 2023, then you’re probably already behind. The truth is that between all of the various economic indicators and investor letters, everyone is looking to improve audit all parts of their business. Especially where there has likely been bloat. One of those… Read more The post Reducing Data Analytics Costs In 2023 – Doing More With Less appeared first on Seattle Data Guy.
KDnuggets
DECEMBER 13, 2022
The cheat sheets cover various NLP techniques, tasks, algorithms, frameworks, and analytics.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Data Engineering Podcast
DECEMBER 11, 2022
Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. Planetscale is a serverless option for your MySQL workloads that lets you focus on your applications without having to worry about managing the database or fight with differences between development and production.
Cloudera
DECEMBER 16, 2022
We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Cloudera has been recognized in this cloud DBMS report since its inception in 2020. This year we’ve been named a Leader. This validates our significant momentum in global enterprises. And together, with our recent recognition in the Gartner Peer Insights Customer Choice Distinction for Cloud DBMS , cements our position as an industry leader.
KDnuggets
DECEMBER 13, 2022
Master the basics of python with this free crash course.
Confluent
DECEMBER 14, 2022
As businesses move to meet modern demands, these technologies ensure not only a digital transformation, but data transformation, with new use cases surrounding real-time data.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Data Engineering Podcast
DECEMBER 11, 2022
Preamble This is a cross-over episode from our new show The Machine Learning Podcast , the show about going from idea to production with machine learning. Summary Data is one of the core ingredients for machine learning, but the format in which it is understandable to humans is not a useful representation for models. Embedding vectors are a way to structure data in a way that is native to how models interpret and manipulate information.
Cloudera
DECEMBER 14, 2022
OCBC Bank optimizes customer experience & risk management with multi-phased data initiative. OCBC Bank is the second largest financial services group in Southeast Asia by assets and one of the most highly-rated banks in the world. Recognised for its financial strength and stability, OCBC Bank is consistently ranked among the World’s Top 50 Safest Banks by Global Finance.
KDnuggets
DECEMBER 14, 2022
Kickstart your career as an MLOps Engineer with this study roadmap.
LinkedIn Engineering
DECEMBER 16, 2022
Saira joined our Bangalore site reliability engineering (SRE) team to tackle large-scale, site engineering challenges and grow. She highlights for us the impactful work she found here �����from ushering in LinkedIn���s next-generation, server query system that runs over a fleet of 350,000 servers, to mentoring the next generation of female engineers: In my engineering career, I���ve always followed the path less taken.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Acceldata
DECEMBER 16, 2022
Data quality is an ever-present issue. But with the right approach, it’s possible to identify data quality problems before they impact your business.
Cloudera
DECEMBER 15, 2022
Since we announced the general availability of Apache Iceberg in Cloudera Data Platform (CDP), Cloudera customers, such as Teranet , have built open lakehouses to future-proof their data platforms for all their analytical workloads. Cloudera partners are also benefiting from Apache Iceberg in CDP. For example, Modak Nabu is helping their enterprise customers accelerate data ingestion, curation, and consumption at petabyte scale.
KDnuggets
DECEMBER 15, 2022
Get more experience by working on web scraping, data analytics, time-series forecasting, machine learning, and deep learning projects.
LinkedIn Engineering
DECEMBER 15, 2022
As teams and applications experience growth, it���s critical to adopt architectures that optimize for clear code ownership, build isolation, and provide efficient delivery of code. While many projects start small with just one or two repositories (for example, frontend and backend), this approach often becomes difficult to maintain as the codebases expand.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
ThoughtSpot
DECEMBER 16, 2022
The FIFA World Cup 2022 is nearing its end, and the final game promises to be a nail biter. What started with 32 countries battling it out for close to a month, will now culminate in a play-off between Argentina and France. FIFA projects more than 5 billion people to tune in for the tournament, perhaps making the World Cup Final 2022 the most watched event of the year!
Cloudera
DECEMBER 13, 2022
How Agencies Can Gain the Cyber Edge with Smart Data Solutions. For the vast majority of US citizens, the front lines of conflict are witnessed from thousands of miles away on the nightly news. But for government agencies, these physical conflicts are the tip of the iceberg as cyberattacks persist as an underlying constant, inflicting enduring damage regardless of geopolitical tension or location. .
KDnuggets
DECEMBER 14, 2022
Markdown is a lightweight markup language for creating formatted text using a plain-text editor. Grab this handy reference sheet to make certain you know how to implement what you need to, when you want to!
LinkedIn Engineering
DECEMBER 13, 2022
Co-authors:�� Kenneth Tay and Xiaofeng Wang At Linkedin, we constantly evaluate the value our products and services deliver, so that we can provide the best possible experiences for our members and customers. This includes understanding how product changes impact key metrics related to those experiences. However, simply looking at connections between product changes and key metrics can be misleading.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Lyft Engineering
DECEMBER 16, 2022
The disproportionate impact of policing on communities of color¹ is a central social and policy concern in the United States, and a topic of intense study in academia. Lyft is uniquely positioned to contribute to this discourse and the academic research and literature on this topic using data from the large number of trips on our rideshare platform.
U-Next
DECEMBER 16, 2022
Exploring the unknown and achieving new milestones every other day seems to be the norm of the 21 st century. Even at the peak of technological innovation and yet hungry to discover and innovate things never heard of before, our determination does not seem to be even mildly deterred by a global pandemic. In fact, the pandemic only made us realize how much we do not know about the world we live in and how much more there is to know and discover.
KDnuggets
DECEMBER 16, 2022
How you can train a model to learn and predict unseen data?
ThoughtSpot
DECEMBER 15, 2022
Late last June, I had the opportunity to attend a ThoughtSpot User Group session in London and share how ThoughtSpot has impacted global procurement at Roche—notably how it’s helped my team not become a dashboard factory for the rest of our business. For 125 years, Roche has been making a difference in the lives of millions. Based out of Basel, Switzerland, we’re a multinational healthcare company with 100,000 plus employees that operates worldwide under two divisions: Pharmaceuticals and Diagno
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Acceldata
DECEMBER 15, 2022
Data observability can data help banks and other finserv players drive growth and profit margins with data quality and data reliability.
databricks
DECEMBER 15, 2022
In Part I of this series, we walked through the process of setting up a Cybersecurity Lakehouse that allowed us to collect and.
KDnuggets
DECEMBER 15, 2022
Choosing the right optimizer to minimize the loss between the predictions and the ground truth is one of the crucial elements of designing neural networks.
U-Next
DECEMBER 15, 2022
Exploring the unknown and achieving new milestones every other day seems to be the norm of the 21 st century. Even at the peak of technological innovation the human’s hunger or determination to discover and innovate things never heard of before does not seem to be even mildly deterred even by a global pandemic. In fact, the pandemic only made us realize how much we do not know about the world we live in and how much more there is to know and discover.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
195 
Let's personalize your content