Confessions of a Data Guy
JUNE 20, 2024
After 10 years of Data Engineering work, I think it’s time to hang up the proverbial hat and ride off into the sunset, never to be seen again. I wish. Everything has changed in 10 years, yet nothing has changed in 10 years, how is that even possible? Sometimes I wonder if I’ve learned anything […] The post What I’ve Learned After A Decade Of Data Engineering appeared first on Confessions of a Data Guy.
KDnuggets
JUNE 21, 2024
Want to learn AI from the best of resources? Check out these free AI courses from top universities.
Christophe Blefari
JUNE 21, 2024
Welcome to the snow world ( credits ) Every year, the competition between Snowflake and Databricks intensifies, using their annual conferences as a platform for demonstrating their power. This year, the Snowflake Summit was held in San Francisco from June 2 to 5, while the Databricks Data+AI Summit took place 5 days later, from June 10 to 13, also in San Francisco.
Data Engineering Podcast
JUNE 16, 2024
Summary Stripe is a company that relies on data to power their products and business. To support that functionality they have invested in Trino and Iceberg for their analytical workloads. In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform.
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
Rockset
JUNE 21, 2024
I’m excited to share that OpenAI has completed the acquisition of Rockset. We are thrilled to join the OpenAI team and bring our technology and expertise to building safe and beneficial AGI. From the start, our vision at Rockset was to fundamentally transform the way data-driven applications were built. We developed our search and analytics database, taking full advantage of the cloud, to eliminate the complexity inherent in the data infrastructure needed for these apps.
KDnuggets
JUNE 20, 2024
Image by author Model deployment is the process of trained models being integrated into practical applications. This includes defining the necessary environment, specifying how input data is introduced into the model and the output produced, and the capacity to analyze new data and provide relevant predictions or categorizations.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Snowflake
JUNE 21, 2024
Thousands of customers have worked with Snowflake to cost-effectively build a secure data foundation as they look to solve a growing variety of business problems with more data. Increasingly customers are looking to expand that powerful foundation to a broader set of data across their enterprise. Snowflake is now making it even easier for customers to bring the platform’s usability, performance, governance and many workloads to more data with Iceberg tables (now generally available), unlocking f
Cloudera
JUNE 17, 2024
Pride Month is underway and we at Cloudera are looking forward to joining the global celebration of diversity, equity and the ongoing effort for LGBTQ+ ( L esbian, G ay, B isexual, T ransgender, Q ueer/ Q uestioning) rights and recognition. Pride Month serves as a reminder that the fight for equality and equity for members of the LGBTQ+ community is not over.
KDnuggets
JUNE 20, 2024
Understanding LLMs is pivotal in unlocking the full potential of AI-driven solutions across various domains. As we navigate the process of building AI-driven solutions, it is essential to approach the development and deployment of LLMs with a focus on responsible AI practices.
databricks
JUNE 20, 2024
We are excited to announce that Gartner has recognized Databricks as a Leader in the 2024 Gartner® Magic Quadrant™ for Data Science and.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Snowflake
JUNE 20, 2024
The telecom industry is undergoing an unprecedented transformation. Fueled by tech advancements such as 5G, cloud computing, Internet of Things (IoT) and machine learning (ML), telecoms have the opportunity to reshape and streamline operations and make significant improvements in service delivery, customer experience and network optimization. Key to these technologies is generative AI (gen AI), a dynamic form of artificial intelligence that leverages vast amounts of data to analyze and produce r
ArcGIS
JUNE 17, 2024
Productivity Update! Learn how to override default parameter values for geoprocessing tools in ArcGIS Pro 3.3. Override Geoprocessing Tool Defaults in ArcGIS Pro 3.
KDnuggets
JUNE 21, 2024
Generating a ready-to-use HuggingFace model with FastAPI and Docker
databricks
JUNE 21, 2024
In the insurance sector, customers demand personalized, fast, and efficient service that addresses their needs. Meanwhile, insurance agents must access a large amount.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Confluent
JUNE 17, 2024
Improve REST API response data w/Kafka and Flink SQL in Confluent Cloud; Automatic connector retriability combats REST flakiness; Demo w/OpenSky data.
Data Engineering Weekly
JUNE 16, 2024
Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Databricks: Open Sourcing Unity Catalog This week brought many exciting developments, with Snowflake and Databricks announcing open-source catalogs.
KDnuggets
JUNE 19, 2024
Perform data integrity tests and generate model evaluation reports by writing a few lines of code.
Engineering at Meta
JUNE 19, 2024
We’re introducing parameter vulnerability factor (PVF) , a novel metric for understanding and measuring AI systems’ vulnerability against silent data corruptions (SDCs) in model parameters. PVF can be tailored to different AI models and tasks, adapted to different hardware faults, and even extended to the training phase of AI models. We’re sharing results of our own case studies using PVF to measure the impact of SDCs in model parameters, as well as potential methods of identifying SDCs in model
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Snowflake
JUNE 20, 2024
In the age of AI, enterprises are increasingly looking to extract value from their data at scale but often find it difficult to establish a scalable data engineering foundation that can process the large amounts of data required to build or improve models. Designed for processing large data sets, Spark has been a popular solution, yet it is one that can be challenging to manage, especially for users who are new to big data processing or distributed systems.
Cloudera
JUNE 21, 2024
Juneteenth holds profound significance in the history of freedom and equality for Black Americans. Also known as Freedom Day or Emancipation Day, Juneteenth commemorates the anniversary of June 19, 1865, when news of the Emancipation Proclamation reached Galveston, Texas, finally declaring freedom for enslaved Americans held in the Confederacy–more than two years after the proclamation was issued in on January 1, 1863.
KDnuggets
JUNE 17, 2024
Boost your data science project with these templates.
Netflix Tech
JUNE 20, 2024
A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale. Netflix is not the only place where data engineers are solving challenging problems with creative solutions.
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Snowflake
JUNE 20, 2024
Recent Snowflake workshops and roundtables have started with the question: “Does your data strategy match your AI ambition?” It certainly sparks customer engagement, but is that the right question to ask? Right now, it seems appropriate with all of the interest — dare I say “hype” — around AI. But it merely reflects the current darling of the tech world, focusing on the technology itself, rather than the ultimate goal.
ArcGIS
JUNE 17, 2024
Discover what's new in ArcGIS Pro 3.3 for CAD and BIM workflows, allowing you to directly read datasets from Autodesk Revit, Civil 3D, and Industry Foundation Classes.
KDnuggets
JUNE 17, 2024
Exploring the available text Python packages for your data workflow.
Cloudera
JUNE 18, 2024
The world of Generative AI (GenAI) is rapidly evolving, with a wide array of models available for businesses to leverage. These models can be broadly categorized into two types: closed-source (proprietary) and open-source models. Closed-source models, such as OpenAI’s GPT-4o, Anthropic’s Claude 3, or Google’s Gemini 1.5 Pro, are developed and maintained by private and public companies.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Precisely
JUNE 19, 2024
Key Takeaways: Adopting both CCM and CRM platforms can significantly enhance your customer experience through personalized communications, automated workflows, and consistent messaging across channels. Automating repetitive communication tasks cuts back on manual efforts to save you time and reduce costs. As consumer expectations for efficient and personalized experiences continue to rise, effectively managing customer relationships and communications is more crucial than ever.
ArcGIS
JUNE 17, 2024
Discover what's new in ArcGIS Pro 3.3 for CAD and BIM workflows, allowing you to directly read datasets from Autodesk Revit, Civil 3D, and Industry Foundation Classes.
KDnuggets
JUNE 19, 2024
Learn about enhancing LLMs with real-time information retrieval and intelligent agents.
DataKitchen
JUNE 21, 2024
Navigating the Storm: How Data Engineering Teams Can Overcome a Data Quality Crisis Ah, the data quality crisis. It’s that moment when your carefully crafted data pipelines start spewing out numbers that make as much sense as a cat trying to bark. You know you’re in trouble when the finance team uses your reports as modern art installations rather than decision-making tools.
Advertisement
Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.
Expert insights. Personalized for you.
223 
Let's personalize your content