KDnuggets
OCTOBER 7, 2022
There is no shortage of tools today that can help you through the steps of natural language processing, but if you want to get a handle on the basics this is a good place to start. Read about the ABCs of NLP, all the way from A to Z.
Jesse Anderson
OCTOBER 5, 2022
My guest this week is Brent Dykes , Founder and Chief Data Storyteller at Analytics Hero. Before he founded his own company, he was at Omniture, Adobe, and Domo. Analytics Hero is a consulting business based around data storytelling Data storytelling was a new concept to me. Brent defines it as “as a structured approach for communicating insights to a targeted audience using narrative elements and explanatory visuals.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Confluent
OCTOBER 3, 2022
Apache Kafka 3.3 includes KRaft mode, improves partition scalability and resiliency while simplifying Kafka deployment, as well as updates to Kafka Streams, Connect, and more.
Data Engineering Podcast
OCTOBER 2, 2022
Summary The global economy is dependent on complex and dynamic networks of supply chains powered by sophisticated logistics. This requires a significant amount of data to track shipments and operational characteristics of materials and goods. Roambee is a platform that collects, integrates, and analyzes all of that information to provide companies with the critical insights that businesses need to stay running, especially in a time of such constant change.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
KDnuggets
OCTOBER 4, 2022
Among the four big NoSQL database types, key-value stores are probably the most popular ones due to their simplicity and fast performance. Let’s further explore how key-value stores work and what are their practical uses.
Cloudera
OCTOBER 3, 2022
Tell us if this sounds familiar. You’ve found an awesome data set that you think will allow you to train a machine learning (ML) model that will accomplish the project goals; the only problem is the data is too big to fit in the compute environment that you’re using. In the day and age of “big data,” most might think this issue is trivial, but like anything in the world of data science things are hardly ever as straightforward as they seem. .
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Data Engineering Podcast
OCTOBER 2, 2022
Summary Data lineage is something that has grown from a convenient feature to a critical need as data systems have grown in scale, complexity, and centrality to business. Alvin is a platform that aims to provide a low effort solution for data lineage capabilities focused on simplifying the work of data engineers. In this episode co-founder Martin Sahlen explains the impact that easy access to lineage information can have on the work of data engineers and analysts, and how he and his team have de
Cloudera
OCTOBER 6, 2022
Digital transformation. Everyone has their own ideas about what digital transformation means, so I decided to look up a few definitions. . Gartner : “Digital transformation can refer to anything from IT modernization (for example, cloud computing), to digital optimization, to the invention of new digital business models.”. CIO blog post : “Digital transformation is a foundational change in how an organization delivers value to its customers.”.
Confluent
OCTOBER 4, 2022
Confluent’s new Stream Designer is the industry’s first visual interface for rapidly building, testing, and deploying streaming data pipelines natively on Apache Kafka.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Teradata
OCTOBER 7, 2022
There are various approaches to doing time-series forecasting. Amongst all the approaches, the right way is using an in-database approach. Read more to find out why.
KDnuggets
OCTOBER 5, 2022
What is NLP, and what types of questions related to NLP can you expect at the NLP-related job interviews?
Cloudera
OCTOBER 7, 2022
Introduction. dbt allows data teams to produce trusted data sets for reporting, ML modeling, and operational workflows using SQL, with a simple workflow that follows software engineering best practices like modularity, portability, and continuous integration/continuous development (CI/CD). We’re excited to announce the general availability of the open source adapters for dbt for all the engines in CDP — Apache Hive , Apache Impala , and Apache Spark, with added support for Apache Livy and Cloude
DataKitchen
OCTOBER 5, 2022
Part 1: Defining the Problems. This is the first post in DataKitchen’s four-part series on DataOps Observability. Observability is a methodology for providing visibility of every journey that data takes from source to customer value across every tool, environment, data store, team, and customer so that problems are detected and addressed immediately.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Confluent
OCTOBER 3, 2022
Announcing Confluent for Startups! Get started with Apache Kafka, leverage our data streaming expertise, and set your business up with the best infrastructure for scale and success.
KDnuggets
OCTOBER 5, 2022
A comprehensive guide on optimizing model hyperparameters with Scikit-Learn.
Cloudera
OCTOBER 4, 2022
This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes. This creates flexibility to handle real-time data feed volumes as they fluctuate.
phData: Data Engineering
OCTOBER 7, 2022
So you’ve built your first set of transformations in dbt , but now you need to figure out how to automate your deployment and code changes to your various environments. However, you’re not sure where to even start planning, let alone making sure that you’re sticking to best practices (whether that’s running your code on a schedule or having it run based on certain actions within your git repository).
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Confluent
OCTOBER 7, 2022
From real-time banking and mobile payments, learn how Apache Kafka and Confluent are powering the financial services industry with event-driven architecture for modern use cases.
KDnuggets
OCTOBER 5, 2022
Digital transformation is evolving, and so is the fintech industry by implementing AI trends and leveraging several benefits, such as optimizing productivity, increasing ROI, and enhancing security.
Cloudera
OCTOBER 1, 2022
While the word “data” has been common since the 1940s, managing data’s growth, current use, and regulation is a relatively new frontier. . Governments and enterprises are working hard today to figure out the structures and regulations needed around data collection and use. According to Gartner, by 2023 65% of the world’s population will have their personal data covered under modern privacy regulations. .
Rockset
OCTOBER 6, 2022
Open source PyTorch runs tens of thousands of tests on multiple platforms and compilers to validate every change as our CI (Continuous Integration). We track stats on our CI system to power custom infrastructure, such as dynamically sharding test jobs across different machines developer-facing dashboards, see hud.pytorch.org , to track the greenness of every change metrics, see hud.pytorch.org/metrics , to track the health of our CI in terms of reliability and time-to-signal Our requirements for
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Emeritus
OCTOBER 6, 2022
Everyone leveraged data from small-scale enterprises to Fortune 500 companies to ensure efficient operations. Frontrunners like the MAANG companies (Meta, Amazon, Apple, Netflix, and Google), have vast databases that hold a wide range of customer data. Here is where database management systems like MS Access come in, and a programming language known as Structured Query… The post What is SQL?
KDnuggets
OCTOBER 7, 2022
We have compiled a list of the top free resources to help new data practitioners learn SQL. These include free online courses and resources to get the most out of your SQL skills.
Monte Carlo
OCTOBER 6, 2022
Will Larson (writer of An Elegant Puzzle – recommended read) may have said it best when he wrote that one of the best kinds of reorganization is the one you don’t do. However, data leaders inevitably reach a point where, due to team growth or evolving business demands, things just don’t work. Faced with these challenges, data organizations may swing back-and-forth between centralized vs. decentralized organizational structures until they achieve the right balance.
Meltano
OCTOBER 5, 2022
Interested in becoming a data engineer? The need for data experts in the U.S. job market is expected to grow by 22% in this decade, and according to LinkedIn’s 2020 report , a data engineer is listed as the 8th fastest growing job today. But what is data engineering exactly and what does a data engineer do? Interested in a data platform? We’ll explain what a data engineer is, what the job entails, and how to become a data engineer.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Robinhood
OCTOBER 5, 2022
Robinhood was founded on a simple idea: that our financial markets should be accessible to all. With customers at the heart of our decisions, Robinhood is lowering barriers and providing greater access to financial information and investing. Together, we are building products and services that help create a financial system everyone can participate in.
KDnuggets
OCTOBER 6, 2022
This article is about 3 ways you can process a CSV file using Python.
Picnic Engineering
OCTOBER 5, 2022
Picnic loves Error Prone: producing high-quality and consistent Java code Error Prone Support is now open source! Check out the announcement. Picnic is changing the way people do groceries. We’re an app-only supermarket, delivering the highest quality products for the best price to our customers. To do this, we meticulously design and build products for our customers and internal users.
Meltano
OCTOBER 5, 2022
Data is becoming the world’s most valuable resource, according to an article in The Economist dating back to 2017. Since then, the way we compile, process, and store data has evolved significantly, and it continues to do so at incredible speed. As more data becomes available, the demand for faster, improved, error-free analytics grows. Who exactly is challenged with meeting this demand?
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
160 
Let's personalize your content