Confessions of a Data Guy
MAY 30, 2024
Of all the duties that Data Engineers take on during the regular humdrum of business and work, it’s usually filled with the same old, same old. Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s never-ending. It’s a constant stream of data, new and old, spilling into our Data Warehouses and […] The post Building Data Platforms (from scratch) appeared first on Confessions of a Data Guy.
Start Data Engineering
MAY 30, 2024
Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct the database to process it [Data Quality] Define what you expect of your data and check if your data confirms it [Code Testing] Ensure your code does
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Waitingforcode
MAY 30, 2024
Last week I was speaking in Gdansk on the DataMass track at Infoshare. As it often happens, the talk time slot impacted what I wanted to share but maybe it's for good. Otherwise, you wouldn't read stream processing fallacies!
Robinhood
MAY 30, 2024
Robinhood Crypto customers in the United States can now use our API to view crypto market data, manage portfolios and account information, and place crypto orders programmatically Today, we are excited to announce the Robinhood Crypto trading API , ushering in a new era of convenience, efficiency, and strategy for our most seasoned crypto traders. Robinhood Crypto customers in the United States can use our new trading API to set up advanced and automated trading strategies that allow them to st
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
databricks
MAY 30, 2024
Salesforce and Databricks are excited to announce an expanded strategic partnership that delivers a powerful new integration - Salesforce Bring Your Own Model.
Snowflake
MAY 30, 2024
We’re excited to announce today that we’re reinforcing our commitment and deepening our partnership with Sigma with an expanded investment from Snowflake Ventures. Sigma is a leading business intelligence and analytics solution that makes it easy for employees to explore live data, create compelling visualizations and collaborate with colleagues. Sigma allows employees to break free of dashboards and build workflows, powered by write-back to Snowflake through their unique Input Tables capability
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Cloudera
MAY 30, 2024
The financial services industry is undergoing a significant transformation, driven by the need for data-driven insights, digital transformation, and compliance with evolving regulations. In this context, Cloudera and TAI Solutions have partnered to help financial services customers accelerate their data-driven transformation, improve customer centricity, ensure compliance with regulations, enhance risk management, and drive innovation.
KDnuggets
MAY 30, 2024
Explore free and open-source MLOps tools for enhanced data privacy and control over your models and code.
Towards Data Science
MAY 30, 2024
Explore how to build, trigger and parameterize a time-series data pipeline in Azure, accompanied by a step-by-step tutorial Continue reading on Towards Data Science »
KDnuggets
MAY 30, 2024
Let’s go beyond the traditional retail industry and discuss how advanced AI-powered innovations are driving business growth.
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Towards Data Science
MAY 30, 2024
MLOps: Datapipeline Orchestration Dataform 101, Part 2: Provisioning with Least Privilege Access Control A typical positioning of Dataform in a data pipeline [Image by author] This is the concluding part of Dataform 101 showing the fundamentals of setting up Dataform with a focus on its authentication flow. This second part focussed on terraform implementation of the flow explained in part 1.
Precisely
MAY 30, 2024
Key Takeaways: Cloud-based High Availability Disaster Recovery (HA-DR) solutions enhance operational efficiency, leveraging automation to streamline recovery processes and reduce downtime expenses. Adopting unique cloud HA-DR strategies improves data redundancy and security, aligns with strict regulatory standards, and proactively manages disaster risks.
Monte Carlo
MAY 30, 2024
Generative AI has the potential to transform your business and your data engineering team, but only when it’s done right. So how can your data team actually drive value with your LLM or GenAI initiative? Leading organizations are often deciding between two emerging frameworks that differentiate their AI for business value: RAG vs fine tuning. What’s the difference between retrieval augmented generation (RAG) vs fine tuning?
RandomTrees
MAY 30, 2024
Creating diagrams has become essential in today’s data-driven world, whether you’re visualizing cloud architectures, documenting processes, or mapping complex systems. Fortunately, a range of powerful tools makes this task easier than ever. From free, open-source options to advanced, collaborative platforms, these tools cater to various needs and preferences.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
databricks
MAY 30, 2024
Whether you are working on a live title, pre/post production, ongoing maintenance, future releases, another version of a game, or a brand new.
Elder Research
MAY 30, 2024
If you’re interested in data science careers, keep reading. This article features some helpful tips for those new to the field.
ArcGIS
MAY 30, 2024
Editing Templates can be stored in the Geodatabase with an Add-In.
Cloudera
MAY 30, 2024
Cloudera customers run some of the biggest data lakes on earth. These lakes power mission-critical, large-scale data analytics and AI use cases—including enterprise data warehouses. Nearly two years ago, Cloudera announced the general availability of Apache Iceberg in the Cloudera platform, which helps users avoid vendor lock-in and implement an open lakehouse.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Expert insights. Personalized for you.
184 
Let's personalize your content