Simon Späti
JANUARY 22, 2022
Image by Mohammad Bagher Adib Behrooz on Unsplash Why GraphQL for data engineers, you might ask? GraphQL solved the problem of providing a distinct interface for each client by unifying it to a single API for all clients such as web, mobile, web apps. The same challenge we’re now facing in the data world, where we integrate multiple clients with numerous backend systems.
Marc Lamberti
JANUARY 9, 2022
Airflow TaskGroups have been introduced to make your DAG visually cleaner and easier to read. They are meant to replace SubDAGs which was the historic way of grouping your tasks. The problem with SubDAGs is that they are much more than that. They bring a lot of complexity as you need to create a DAG in a DAG, import the SubDagOperator which is in fact a sensor, define the parameters properly, and so on.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
JANUARY 24, 2022
What does the data reveal if we ask: "What are the 10 Best Python Courses?". Collecting almost all of the courses from top platforms shows there are plenty to choose from, with over 3000 offerings. This article summarizes my analysis and presents the top three courses.
Confluent
JANUARY 11, 2022
Whether you’re a seasoned Apache Kafka® developer or just getting started you’re likely to hit a snag at some point or another—either in configuring and understanding your clients or setting […].
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Data Engineering Podcast
JANUARY 30, 2022
Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. Matt Harrison is a Python expert with a long history of working with data who now spends his time on consulting and training. He recently wrote a book on effective patterns for Pandas code, and in this episode he shares advice on how to write efficient data processing routines
Cloudera
JANUARY 28, 2022
As I meet with our customers, there are always a range of discussions regarding the use of the cloud for financial services data and analytics. Customers vary widely on the topic of public cloud – what data sources, what use cases are right for public cloud deployments – beyond sandbox, experimentation efforts. Private cloud continues to gain traction with firms realizing the benefits of greater flexibility and dynamic scalability.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
U-Next
JANUARY 17, 2022
At a recent training for fresher hire as part of an MNC’s analytics training program, my colleague Dr. Chetana highlighted that only 10% of the hires were women. TrustRadius reported that in 2021, 72% of women in tech are outnumbered by men in business meetings by at least a 2:1 ratio. Women are less than 1/3rd of the employees in many tech companies.
KDnuggets
JANUARY 15, 2022
In our first weekly roundup of data science nuggets from around the web, check out a list of curated articles on Kaggle datasets, Python debugging tools, what it is data scientists do, an overview of YOLO, 2-dimensional PyTorch tensors, and the secrets of machine learning deployment.
Confluent
JANUARY 5, 2022
Chances are your business is migrating to the cloud. But if you operate business applications in an on-premises datacenter, you know firsthand that the journey to the cloud is fraught […].
Data Engineering Podcast
JANUARY 30, 2022
Summary The Data Engineering Podcast has been going for five years now and has included conversations and interviews with a huge number of guests, covering a broad range of topics. In addition to that, the host curated the essays contained in the book "97 Things Every Data Engineer Should Know", using the knowledge and context gained from running the show to inform the selection process.
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Cloudera
JANUARY 21, 2022
This blog will summarise the security architecture of a CDP Private Cloud Base cluster. The architecture reflects the four pillars of security engineering best practice, Perimeter, Data, Access and Visibility. The release of CDP Private Cloud Base has seen a number of significant enhancements to the security architecture including: Apache Ranger for security policy management.
DataKitchen
JANUARY 25, 2022
Analytics are prone to frequent data errors and deployment of analytics is slow and laborious. The strategic value of analytics is widely recognized, but the turnaround time of analytics teams typically can’t support the decision-making needs of executives coping with fast-paced market conditions. Perhaps it is no surprise that the average tenure of a CDO or CAO is only about 2.5 years.
U-Next
JANUARY 17, 2022
Introduction. The evolution of workplaces has seen people being hired for more than just their educational qualifications. The criteria for being hired has seen a tremendous shift in the digital age. Along with skill and knowledge in the necessary domain, companies are keen on hiring professionals with strong critical thinking capabilities. This ensures that the employees are able to deal with real-time issues with a practical approach. .
KDnuggets
JANUARY 24, 2022
There are many great boosting Python libraries for data scientists to reap the benefits of. In this article, the author discusses LightGBM benefits and how they are specific to your data science job.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Confluent
JANUARY 24, 2022
On behalf of the Apache Kafka® community, it is my pleasure to announce the release of Apache Kafka 3.1.0. The 3.1.0 release contains many improvements and new features. We’ll highlight […].
Data Engineering Podcast
JANUARY 23, 2022
Summary Data platforms are exemplified by a complex set of connections that are subject to a set of constantly evolving requirements. In order to make this a tractable problem it is necessary to define boundaries for communication between concerns, which brings with it the need to establish interface contracts for communicating across those boundaries.
Cloudera
JANUARY 18, 2022
Each year, the Cloudera Data Impact Awards recognize organizations that have accomplished amazing things with innovative data solutions. . For 2021, the awards will include a new category: People First. Entrants in this category were asked to demonstrate how they have addressed the world’s “most difficult workplace and societal challenges” with solutions aimed at transforming work culture and society as a whole.
DataKitchen
JANUARY 3, 2022
Business analysts often find themselves in a no-win situation with constraints imposed from all sides. Their business unit colleagues ask an endless stream of urgent questions that require analytic insights. Business analysts must rapidly deliver value and simultaneously manage fragile and error-prone analytics production pipelines. Data tables from IT and other data sources require a large amount of repetitive, manual work to be used in analytics.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
U-Next
JANUARY 17, 2022
A couple of decades ago, ‘Data’ was analyzed manually. With the advent of data management tools, we were able to computerize the same ‘Data’ for deeper analysis. Thus the trend of driving business decisions via insights drawn from data sets has never been old. However, with the availability of tools to manage and analyze data, the quantity and the quality of data analyzed have improved drastically, thereby increasing the accuracy and the efficacy of data-driven decisions.
KDnuggets
JANUARY 21, 2022
Just like tradespeople need to grow in their skill sets, data scientists must also grow in the ever-changing world we inhabit. With that said, let’s break down how you can evolve your data science skills while progressing your career.
Confluent
JANUARY 6, 2022
While Self-Balancing Clusters (SBC) perform effectively in balancing Apache Kafka® clusters, one of the common themes we hear from our users is that they would love some visibility into the […].
Data Engineering Podcast
JANUARY 23, 2022
Summary Data engineering is a relatively young and rapidly expanding field, with practitioners having a wide array of experiences as they navigate their careers. Ashish Mrig currently leads the data analytics platform for Wayfair, as well as running a local data engineering meetup. In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
Teradata
JANUARY 10, 2022
There is a lot of hype today around data sharing and the value it brings to your business. But what exactly is data sharing, and why should you and your company care? Find out more.
Netflix Tech
JANUARY 13, 2022
By Vikram Srivastava and Marcelo Mayworm Netflix has one of the most complex data platforms in the cloud on which our data scientists and engineers run batch and streaming workloads. As our subscribers grow worldwide and Netflix enters the world of gaming , the number of batch workflows and real-time data pipelines increases rapidly. The data platform is built on top of several distributed systems, and due to the inherent nature of these systems, it is inevitable that these workloads run into fa
U-Next
JANUARY 17, 2022
A domain with applications across multiple industries from Agriculture to Transport, Business Analytics is all about making data-driven decisions for maximum business revenue. Even though this field has established a strong presence over the years, there’s an array of opportunities and growth still waiting to be transformed into reality. . According to IMARC Group’s latest report , the global BPO business analytics market is expected to grow at a CAGR of around 25% during 2021-2026.
KDnuggets
JANUARY 17, 2022
In this article, Eric Siegel summarizes the recent KDnuggets poll results and argues that the pervasive failure of ML projects comes from a lack of prudent leadership. He also argues that MLops is not the fundamental missing ingredient – instead, an effective ML leadership practice must be the dog that wags the model-integration tail.
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
Confluent
JANUARY 12, 2022
We’re pleased to announce ksqlDB 0.23.1! This release allows you to now perform pull queries on streams, which makes it much easier to find a given record in a topic. […].
Data Engineering Podcast
JANUARY 15, 2022
Summary Applications of data have grown well beyond the venerable business intelligence dashboards that organizations have relied on for decades. Now it is being used to power consumer facing services, influence organizational behaviors, and build sophisticated machine learning systems. Given this increased level of importance it has become necessary for everyone in the business to treat data as a product in the same way that software applications have driven the early 2000s.
Cloudera
JANUARY 19, 2022
Gartner® recognized Cloudera in three recent reports – Magic Quadrant for Cloud Database Management Systems (DBMS), Critical Capabilities for Cloud Database Management Systems for Analytical Use Cases and Critical Capabilities for Cloud Database Management Systems for Operational Use Cases. Our position as a Visionary in the Gartner Magic Quadrant for Cloud DBMS market speaks to our product excellence and market-leading-vision of a hybrid, multifunction integrated platform with built-in security
Teradata
JANUARY 31, 2022
From Open Finance and Insurance to FinCrime and Crypto, hear from one of our expert on the top FinServe trends and predictions to look out for in 2022. Read more.
Advertisement
Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.
Expert insights. Personalized for you.
147 
Let's personalize your content