Start Data Engineering
MAY 18, 2022
1. Introduction 2. Set up 3. Reproducibility 3.1. Docker 3.2. Docker Compose 4. Developer ergonomics 4.1. Formatting and testing 4.2. Makefile 5. Conclusion 6. Further reading 7. References 1. Introduction Data systems usually involve multiple systems, which makes local development challenging.
Azure Data Engineering
MAY 15, 2022
When it comes to transforming structured data, (e.g., applying business logic, standardization etc.) stored in a database, SQL is the most convenient and fit-to-purpose option. Stored procedures provide a way to store the transformation logic as a set of SQL statements that can be re-executed as pre-compiled code. The Stored Procedure Activity in Data Factory provides and simple and convenient way to execute Stored Procedures.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
MAY 19, 2022
Read the best books on Programming, Statistics, Data Engineering, Web Scraping, Data Analytics, Business Intelligence, Data Applications, Data Management, Big Data, and Cloud Architecture.
Confluent
MAY 17, 2022
I’m proud to announce the release of Apache Kafka 3.2.0 on behalf of the Apache Kafka® community. The 3.2.0 release contains many new features and improvements. This blog will highlight […].
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Data Engineering Podcast
MAY 15, 2022
Summary Industrial applications are one of the primary adopters of Internet of Things (IoT) technologies, with business critical operations being informed by data collected across a fleet of sensors. Vopak is a business that manages storage and distribution of a variety of liquids that are critical to the modern world, and they have recently launched a new platform to gain more utility from their industrial sensors.
Cloudera
MAY 18, 2022
Deciding to adopt an AI-first strategy is the easy part. Figuring out how to implement it takes a little more effort. It requires a clear-eyed vision built around well-defined goals and a realistic execution plan. Being AI-first means setting up your organization for the future. By leveraging data, analytics, and automation, a company can gain a better understanding of where it is and where it needs to go.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Confluent
MAY 18, 2022
There’s an increasing need for businesses to act intelligently and in real time to win in today’s digital-first world. To achieve this, forward-thinking companies are modernizing their data infrastructure with […].
Data Engineering Podcast
MAY 15, 2022
Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience.
Cloudera
MAY 19, 2022
From fashion to data flow, in this #ClouderaLife Spotlight Margot talks about her career transition from fashion design to cloud computing and her co-founding of Cloudera’s Asian American and Pacific Islander community Employee Resource Group amid the racial tensions of 2021. . It started with feeling stuck and ended with a brand-new career (BTW, lots of hard work in the middle).
KDnuggets
MAY 16, 2022
This post provides a concise overview of 18 natural language processing terms, intended as an entry point for the beginner looking for some orientation on the topic.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Teradata
MAY 19, 2022
Faced with persistent supply chain disruption automotive companies need a new approach to planning. Find out more.
dbt Developer Hub
MAY 18, 2022
If you're reading this article, it looks like you're wondering how you can better optimize your Redshift queries - and you're probably wondering how you can do that in conjunction with dbt. In order to properly optimize, we need to understand why we might be seeing issues with our performance and how we can fix these with dbt sort and dist configurations.
Confluent
MAY 18, 2022
With more data being produced in real time by many systems and devices than ever before, it is critical to be able to process it in real time and get […].
KDnuggets
MAY 18, 2022
Complete guide and blog post series on IT Operations Management with AIOps. Using AI and Machine Learning to manage IT complexity to deliver world class IT service while keeping the lights on.
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Emeritus
MAY 19, 2022
With businesses increasingly relying on data for their day-to-day operations, the role of a data engineer has emerged as one of the most sought-after professions in the industry. But what does a data engineer do exactly? And why is it in demand? According to McKinsey, by 2025, smart workflows and seamless interactions between humans and… The post What Does a Data Engineer do?
Big Data Tools
MAY 19, 2022
Long time no see! Sorry about the silence, but luckily we’re back. Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see.
Rockset
MAY 17, 2022
This is the fourth post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them! Posts published so far in the series: Why Mutability Is Essential for Real-Time Data Analytics Handling Out-of-Order Data in Real-Time Analytics Applications Handling Bursty Traffic in Real-Time Analytics Applications SQL and Complex Queries
KDnuggets
MAY 20, 2022
Most companies haven’t seen ROI from machine learning since the benefit is only realized when the models are in production. Here’s how to make sure your ML project works.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Emeritus
MAY 19, 2022
Data is so ubiquitous and valuable that it is touted as the new currency. From data analytics to data engineering, everything is data-centric. As Carly Fiorina, the former Chief Executive Officer of Hewlett Packard, said, “The goal is to turn data into information, and information into insight.” Data allows leaders to make informed decisions that… The post What Does a Data Engineer Do and How Can You Become One?
Big Data Tools
MAY 19, 2022
Long time no see! Sorry about the silence, but luckily we’re back. Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see.
Monte Carlo
MAY 16, 2022
Data lineage isn’t new, but automation has finally made it accessible and scalable—to a certain extent. In the old days (way back in the mid-2010s), lineage happened through a lot of manual work. This involved identifying data assets, tracking them to their ingestion sources, documenting those sources, mapping the path of data as it moved through various pipelines and stages of transformation, and pinpointing where the data was served up in dashboards and reports.
KDnuggets
MAY 20, 2022
We give a taxonomy of the trustworthy GNNs in privacy, robustness, fairness, and explainability. For each aspect, we categorize existing works into various categories, give general frameworks in each category, and more.
Advertisement
Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.
dbt Developer Hub
MAY 16, 2022
Analytics engineers (AEs) are constantly navigating through the names of the models in their project, so naming is important for maintainability in your project in the way you access it and work within it. By default, dbt will use your model file name as the view or table name in the database. But this means the name has a life outside of dbt and supports the many end users who will potentially never know about dbt and where this data came from, but still access the database objects in the datab
Rock the JVM
MAY 16, 2022
Scala's general type projections are considered unsound and were removed in Scala 3: discover what this means and how it affects your code
Monte Carlo
MAY 19, 2022
In his seminal 2017 blog post, The Downfall of the Data Engineer , Maxime Beauchemin wrote that the data engineer had the worst seat at the table. Data technology and teams have changed tremendously since that time, and now the Preset CEO and creator of Apache Airflow and Apache Superset has a brighter outlook on the future of the profession. I have also seen what was once a thankless position turn into a strategic driver of company value as data expanded beyond dashboards to machine learning mo
KDnuggets
MAY 17, 2022
As a data scientist, you might have a great portfolio of technical skills, but if you can’t communicate effectively, you won’t be able to convey your ideas clearly during virtual meetings.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
dbt Developer Hub
MAY 14, 2022
There are so many different date functions in SQL—you have DATEDIFF , DATEADD , DATE_PART, and DATE_TRUNC to name a few. They all have their different use cases and understanding how and when they should be used is a SQL fundamental to get down. Are any of those as easy to use as the EXTRACT function? Well, that debate is for another time… In this post, we’re going to give a deep dive into the EXTRACT function, how it works, and why we use it.
KDnuggets
MAY 20, 2022
Let's look at six must-have tools every data scientist should use.
KDnuggets
MAY 17, 2022
Hugging Face has released a free course on Deep RL. It is self-paced and shares a lot of pointers on theory, tutorials, and hands-on guides.
KDnuggets
MAY 16, 2022
A simple guide to reinforcement learning for a complete beginner. The blog includes definitions with examples, real-life applications, key concepts, and various types of learning resources.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
Expert insights. Personalized for you.
148 
Let's personalize your content