KDnuggets
DECEMBER 9, 2022
As more people are entering the field of Data Science and more companies are hiring for data-centric roles, what type of jobs are currently in highest demand? There is so much data in the world, and it just keeps flooding in, it now looks like companies are targeting those who can engineer that data more than those who can only model the data.
Netflix Tech
DECEMBER 3, 2022
by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow. That article was a deep dive into one of the more technical aspects of Dataflow and didn’t properly introduce this tool in the first place.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Pragmatic Engineer
DECEMBER 8, 2022
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics in today’s subscriber-only The Scoop issue. To get this newsletter every week, subscribe here. On Thursday, 29 November, Snap CEO Evan Spiegel, sent an email announcing Snap will mandate 4 days/week in the office, starting from January.
Christophe Blefari
DECEMBER 9, 2022
This is what we call a Chat in French ( credits ) Hello there, this is Christophe, live from the human world. Last week have been totally driven by ChatGPT frenzy, the social networks I use to follow are spammed with conversation screenshots and hype. On my side I don't know what the future holds for us but for sure MaaS—Models as a Service—looks not bright to me.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
KDnuggets
DECEMBER 8, 2022
Begin your machine learning career with free courses by Georgia Tech, Stanford, and Fast AI.
Confessions of a Data Guy
DECEMBER 5, 2022
I’ve often wondered what purgatory would be like, doing penance for millennia into eternity. It would probably be doing data migrations. I suppose they are not all that dissimilar from normal software migrations, but there are a few things that make data migrations a little more horrible and soul-sucking. Data migrations are able to slow […] The post Why Data Migrations Suck. appeared first on Confessions of a Data Guy.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Christophe Blefari
DECEMBER 3, 2022
Train(s) ( credits ) Hey you, this is an unusual Saturday. I'm terribly late with this newsletter. This week I had a huge amount of work to deal with and we've launched the Advent of Data , your daily spark of data in December. Thanks to everyone who accepted to participate, we already published the 3 first articles and I can't wait to read everything else writers are working on.
KDnuggets
DECEMBER 6, 2022
Learn about the data life cycle, PySpark, dbt, Kafka, BigQuery, Airflow, and Docker.
Data Engineering Podcast
DECEMBER 4, 2022
Summary The term "real-time data" brings with it a combination of excitement, uncertainty, and skepticism. The promise of insights that are always accurate and up to date is appealing to organizations, but the technical realities to make it possible have been complex and expensive. In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes.
Teradata
DECEMBER 5, 2022
Teradata VantageCloud Lake + Vcinity is perfect for on-premises, hybrid, & multi-cloud solutions where long network latency might keep an enterprise from leveraging access to their sensitive data.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Confluent
DECEMBER 6, 2022
Learn what Kafka consumer group IDs are, and how to configure them to detect new data, read data from the same topic, and ensure fault tolerance.
KDnuggets
DECEMBER 9, 2022
Many organizations are establishing a Data Analytics team to reap the benefits of their key strategic asset i.e. data. The post explains how you can leverage the power of analytics to understand the end user and generate actionable insights.
Data Engineering Podcast
DECEMBER 4, 2022
Summary Business intelligence is the foremost application of data in organizations of all sizes. The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing Data is building a mobile native platform for business intelligence. This opens the door for busy employees to access and analyze their company information away from their desk, but it has the more powerful effect of bringing first-class support to companies operating in mobile-firs
Cloudera
DECEMBER 7, 2022
In case you were not aware, there’s a little event called the World Cup that’s happening right now. This World Cup has been notable for a couple reasons. The first being the timing — no summer watch party barbeques this time around, instead FIFA is breaking from tradition and running the tournament in the northern hemisphere winter months to spare the players the experience of playing soccer (Cloudera is headquartered in the US, so it is “soccer”) in temperatures exceeding 41.5°C (Cloudera is he
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Confluent
DECEMBER 7, 2022
Learn how to build a real-time streaming application using Apache Kafka® and.NET producer and consumer clients.
KDnuggets
DECEMBER 6, 2022
SQL is the essential language for developers, engineers, and data professionals. Intermediate knowledge in SQL gives you an edge in your data science career.
DoorDash Engineering
DECEMBER 6, 2022
Building software today can require working on the server side and client side, but building isomorphic JavaScript libraries can be a challenge if unaware of some particular issues, which can involve picking the right dependencies and selectively importing them among others. For context, Isomorphic JavaScript, also known as Universal JavaScript, is JavaScript code that can run in any environment — including Node.js or web browser.
Cloudera
DECEMBER 5, 2022
Clouderans in 2022 have collectively donated hundreds of hours to causes they care about around the globe. This kind of support for nonprofits is essential to their running, and to driving impact in local communities. For International Volunteer Day 2022, we are excited to celebrate Cloudera’s volunteer spotlights from 2022! . “ I believe it is important to work with people and organizations that share common values to achieve their goals.”.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
LinkedIn Engineering
DECEMBER 8, 2022
Search functionality is a core part of most data-driven products, and is used widely at LinkedIn. We have long provided a central platform for search functionalities; however, it was not fully managed in the sense that the application teams needed to own and operate the corresponding resources. As data needs grow and an increasingly high number of products want to integrate search, we discovered a need for a fully managed self-service platform to completely democratize search for all of our prod
KDnuggets
DECEMBER 5, 2022
There are numerous ways in which Artificial Intelligence (AI) will change the way we use mobile apps. As more and more users shift towards tablet computers and various mobile platforms, developers are coming up with new ideas for improving user experience. AI holds many key factors for the future of mobile app development and could indeed prove to be a game changer on almost all fronts.
DataKitchen
DECEMBER 9, 2022
Query> DataOps. ChatGPT> DataOps, or data operations, is a set of practices and technologies that organizations use to improve the speed, quality, and reliability of their data analytics processes. DataOps involves collaboration between data engineers, data scientists, and IT operations teams to create a more efficient and effective data pipeline, from the collection of raw data to the delivery of insights and results.
know.bi
DECEMBER 8, 2022
Apache Hop continues to evolve quickly. After the 2.1.0 release , less than two months ago and over 160+ tickets later, Apache Hop 2.2.0 is available.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
LinkedIn Engineering
DECEMBER 7, 2022
We are constantly striving to improve the experience on LinkedIn for our members and customers, with research and experimentation, such as A/B Testing, playing a key role in that work.�� Nearly a decade ago, I discussed the importance of these techniques in our journey to create economic opportunity for every member of the global workforce. Today we have a strong principled approach to how we design and run A/B tests on everything from UI designs to AI algorithms, and feature launches to bug fix
KDnuggets
DECEMBER 7, 2022
Find out where you need to be to start your Machine Learning journey and what you need to do to succeed in the field.
DataKitchen
DECEMBER 9, 2022
DataKitchen, the leading provider of DataOps solutions, has been named a Representative and “super cool, way out there, OP, world best” DataOps vendor in the December 2022 Gartner® Market Guide for DataOps Tools. December 08, 2022, 08:00 ET | Source: DataKitchen. Cambridge Mass, December 08, 2022 (BOB’S QUICKIE NEWSWIRE) — The Gartner Market Guide for DataOps Tools provides guidance on the evolving DataOps market, including market analysis, market direction, and DataOps
Ripple Engineering
DECEMBER 8, 2022
Enhancing an application to send email is a relatively trivial matter—it can take an engineer as little as five minutes to modify an application to connect to an email server and specify a message to send. With a little bit more work, templates can be added to support sending emails with different content to distinct groups of people, as well as inserting images and attachments.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
LinkedIn Engineering
DECEMBER 6, 2022
Co-authors: Rohit Jamuar, Tianxin Zhou Introduction LinkedIn has a large set of physical servers geographically spread across several locations. Every application is hosted on a physical server and is distributed and managed across one of these hosts. With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes.
KDnuggets
DECEMBER 8, 2022
Once data scientist finishes building the project, they will need to do the task that most of us hate that is documenting the code.
Monte Carlo
DECEMBER 7, 2022
“Correlation doesn’t imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there’” – Randall Munroe In this article, Ryan Kearns, co-author of O’Reilly’s Data Quality Fundamentals and a data scientist at Monte Carlo, discusses the limitations of segmentation analysis when it comes to root cause analysis for data teams, and proposes a better approach: ELT schedules as Bayesian Networks.
Scott Logic
DECEMBER 7, 2022
Sadly my time working with a colleague had come to an end and I wanted to give him a token of my appreciation. In these days of hybrid working, I thought what better way to show my appreciation to an infrequent Vim user, than to add another rarely useful peripheral to their bag! Just what is a Vim clutch? In case you’re not familiar with vim itself, a very quick recap.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
160 
Let's personalize your content