Start Data Engineering
MAY 11, 2022
1. Introduction 2. Objective 3. Design 4. Setup 4.1 Prerequisite 4.2 AWS Infrastructure costs 4.3 Data lake structure 5. Code walkthrough 5.1 Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7. Design considerations 8.
Azure Data Engineering
MAY 28, 2022
ARM or Azure Resource Manager templates make it easy to manage deployments for Data Factory. When we connect Data Factory to a source control repository (e.g. GitHub or Azure DevOps Git), the data factory along with all its artefacts ( pipelines , datasets , linked services etc.) is saved in the repository in the form of ARM templates. We can then create DevOps pipelines to manage deployments by overriding the parameters to deploy to the production environments.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Data Engineering Podcast
MAY 22, 2022
Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager. In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and communi
KDnuggets
MAY 26, 2022
Read the best books on Machine Learning, Deep Learning, Computer Vision, Natural Language Processing, MLOps, Robotics, IoT, AI Products Management, and Data Science for Executives.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Confluent
MAY 17, 2022
I’m proud to announce the release of Apache Kafka 3.2.0 on behalf of the Apache Kafka® community. The 3.2.0 release contains many new features and improvements. This blog will highlight […].
Cloudera
MAY 4, 2022
Artificial intelligence (AI) has been a focus for research for decades, but has only recently become truly viable. The availability and maturity of automated data collection and analysis systems is making it possible for businesses to implement AI across their entire operations to boost efficiency and agility. AI has the potential to transform operations by improving three fundamental business requirements: process automation, decision-making based on data insights, and customer interaction.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Azure Data Engineering
MAY 15, 2022
When it comes to transforming structured data, (e.g., applying business logic, standardization etc.) stored in a database, SQL is the most convenient and fit-to-purpose option. Stored procedures provide a way to store the transformation logic as a set of SQL statements that can be re-executed as pre-compiled code. The Stored Procedure Activity in Data Factory provides and simple and convenient way to execute Stored Procedures.
Data Engineering Podcast
MAY 29, 2022
Summary A large fraction of data engineering work involves moving data from one storage location to another in order to support different access and query patterns. Singlestore aims to cut down on the number of database engines that you need to run so that you can reduce the amount of copying that is required. By supporting fast, in-memory row-based queries and columnar on-disk representation, it lets your transactional and analytical workloads run in the same database.
KDnuggets
MAY 30, 2022
Also: Decision Tree Algorithm, Explained; Data Science Projects That Will Land You The Job in 2022; The 6 Python Machine Learning Tools Every Data Scientist Should Know About; Naïve Bayes Algorithm: Everything You Need to Know.
Confluent
MAY 12, 2022
Internet connectivity is something we sometimes take for granted. For many, most places we visit, work, or reside have some form of connectivity whether it be cellular, Wi-Fi, fiber, etc. […].
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
AltexSoft
MAY 12, 2022
We live in the world of sounds: Pleasant and annoying, low and high, quiet and loud, they impact our mood and our decisions. Our brains are constantly processing sounds to give us important information about our environment. But acoustic signals can tell us even more if analyze them using modern technologies. Today, we have AI and machine learning to extract insights, inaudible to human beings, from speech, voices, snoring, music, industrial and traffic noise, and other types of acoustic signals
Cloudera
MAY 9, 2022
Tuning Hive on Tez queries can never be done in a one-size-fits-all approach. The performance on queries depends on the size of the data, file types, query design, and query patterns. During performance testing, evaluate and validate configuration parameters and any SQL modifications. It is advisable to make one change at a time during performance testing of the workload, and would be best to assess the impact of tuning changes in your development and QA environments before using them in product
Netflix Tech
MAY 21, 2022
At Netflix, we want to entertain the world through creating engaging content and helping members discover the titles they will love. Key to that is understanding causal effects that connect changes we make in the product to indicators of member joy. To measure causal effects we rely heavily on AB testing , but we also leverage quasi-experimentation in cases where AB testing is limited.
Data Engineering Podcast
MAY 29, 2022
Summary The latest generation of data warehouse platforms have brought unprecedented operational simplicity and effectively infinite scale. Along with those benefits, they have also introduced a new consumption model that can lead to incredibly expensive bills at the end of the month. In order to ensure that you can explore and analyze your data without spending money on inefficient queries Mingsheng Hong and Zheng Shao created Bluesky Data.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
KDnuggets
MAY 27, 2022
Colossal amounts of data need to be dealt with by specialists. It’s no wonder then that the job prospects in this industry are expected to rise much faster than in other occupations.
Confluent
MAY 11, 2022
Walmart’s global presence, with its vast number of retail stores plus its robust and rapidly growing e-commerce business, make it one of the most challenging retail companies on the planet […].
AltexSoft
MAY 27, 2022
How many days will a particular person spend in a hospital? Healthcare facilities and insurance companies would give a lot to know the answer for each new admission. Today, we can employ AI technologies to predict the date of discharge. This article describes how data and machine learning help control the length of stay — for the benefit of patients and medical organizations.
Cloudera
MAY 2, 2022
As digital transformation accelerates, and digital commerce increasingly becomes the dominant form of all commerce, regulators and governments around the world are recognizing the increased need for consumer protections and data protection measures. The European Union has been at the vanguard for some time (most recently having reached provisional agreement on the Digital Services Act ) but from Australia to Brazil , from South Africa to California (the rest of the US hasn’t quite caught on yet!
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Teradata
MAY 19, 2022
Faced with persistent supply chain disruption automotive companies need a new approach to planning. Find out more.
Data Engineering Podcast
MAY 22, 2022
Summary The interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. With an eye to making data workflows more accessible to everyone in an organization Raj Bains and his team at Prophecy designed a powerful and extensible low-code platform that lets technical and non-technical users scale data flows without forcing everyone into the same layers of abstraction.
KDnuggets
MAY 19, 2022
Read the best books on Programming, Statistics, Data Engineering, Web Scraping, Data Analytics, Business Intelligence, Data Applications, Data Management, Big Data, and Cloud Architecture.
Confluent
MAY 30, 2022
Kafka is horizontally scalable, but it's not enough. So we made Confluent Cloud 10x more elastic - 10x faster to scale up to GB/s or down to zero, easier to use, and cost-effective.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
dbt Developer Hub
MAY 18, 2022
If you're reading this article, it looks like you're wondering how you can better optimize your Redshift queries - and you're probably wondering how you can do that in conjunction with dbt. In order to properly optimize, we need to understand why we might be seeing issues with our performance and how we can fix these with dbt sort and dist configurations.
Cloudera
MAY 5, 2022
Fraud, waste, and abuse (FWA) in government is a constant, multi-billion dollar issue that challenges agency leaders at all levels and across all sectors, from healthcare to education to taxation to Social Security. The scope and scale of public spending — federal outlays alone were approximately $6.6 trillion in fiscal year 2020 according to the Congressional Budget Office — make FWA an inherently difficult problem to solve.
Teradata
MAY 12, 2022
The next generation of customers expects newer technologies & advanced self-service capabilities as the airline business becomes more competitive. How can airlines meet these expectations?
Data Engineering Podcast
MAY 15, 2022
Summary Industrial applications are one of the primary adopters of Internet of Things (IoT) technologies, with business critical operations being informed by data collected across a fleet of sensors. Vopak is a business that manages storage and distribution of a variety of liquids that are critical to the modern world, and they have recently launched a new platform to gain more utility from their industrial sensors.
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
KDnuggets
MAY 16, 2022
This guide will help aspiring data scientists and machine learning engineers gain better knowledge and experience. I will list different types of machine learning algorithms, which can be used with both Python and R.
Confluent
MAY 30, 2022
What we’ve done to evolve from cloud Kafka to Confluent Cloud, a data streaming platform that’s 10X better than Kafka in elasticity, storage, resiliency, and more.
Monte Carlo
MAY 31, 2022
When a data pipeline breaks, data engineers need to immediately understand where the rupture occurred and what has been impacted. Data downtime is costly. Without data lineage –a map of how assets are connected and data moves across its lifecycle–data engineers might as well conduct their incident triage and root cause analysis blindfolded. Field-level data lineage (not necessarily Spark lineage) with hundreds of connections between objects in upstream and downstream tables.
Cloudera
MAY 18, 2022
Deciding to adopt an AI-first strategy is the easy part. Figuring out how to implement it takes a little more effort. It requires a clear-eyed vision built around well-defined goals and a realistic execution plan. Being AI-first means setting up your organization for the future. By leveraging data, analytics, and automation, a company can gain a better understanding of where it is and where it needs to go.
Advertisement
Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.
Expert insights. Personalized for you.
246 
Let's personalize your content