The data industry has gone through multiple evolutions since it gained speed in 1990s, when a lot of data was still held on analog formats like cassettes. In recent years, the rollout of 5G connectivity and vast IoT networks have given companies more data to harvest than ever before.
While new technologies broaden the scope of information available to big data algorithms, other advances deepen the amount of information that can be gleaned from existing sources. For example, faster CPUs and in-memory processing allow for more responsive, real-time analytics, while the advance of natural language processing technology helps algorithms to read, understand and use information presented in written human language.
Ask your average techie on the street to name the most consequential big data firms, and you’ll likely hear names like IBM, Oracle and Cloudera. While they’re critical players, they only represent the tip of the iceberg. There are plenty of companies based right here in San Francisco who are shifting the ways in which data is collected and leveraged — and these four serve as a solid sampling of the work happening across the industry.
The work: Domino Data Lab built a software platform designed to automate DevOps for data science, deploy code on a Kubernetes grid and manage data science teams and projects. The technology lets users choose and work with their choice of infrastructure tools, languages, visualization tech, machine learning algorithms and deep learning models, then run experiments and compare results within the platform.
Pay attention because: Everyone from music companies to grocery stores to medicine manufacturers now have the potential to benefit from data, and Domino Data Lab’s technology is helping many do just that. While the company isn’t necessarily a household name, its platform helps companies like Lockheed Martin, Dell and Bayer deploy deep learning models.
The work: Sift uses complex machine learning to monitor a diverse set of customer data troves, processing some 35 billion events per month. The technology looks for tell-tale signs of payment fraud, promo abuse, account takeovers and more, essentially acting as an automated anti-fraud platform. The technology comes with case management and reporting tools that users can customize to suit their unique datasets.
Pay attention because: As e-commerce companies and data harvesters explore new ways to monetize online activities, criminal organizations and fraudsters find less savory methods to line their pockets. Examples include fake account creation, credential compromises, stolen credit card use, payment fraud, spam and online scams — all of which can severely damage a brand’s reputation. By automating the laborious work of monitoring and mitigating suspicious activity, Sift acts as a brand’s bulwark against a rising tide of cybercrime.
The work: Crunchbase is a database with details on metrics like funding, mergers, acquisitions, investments, staff and revenue estimates for companies around the world. The company accepts crowdsourced information, partner contributed content and automatically scraped details. Crunchbase then deploys a data management team to validate and enrich the overall data set. Investors, sales reps and journalists use Crunchbase to track the rise and fall of companies they want to invest in, sell to or write about. The technology also powers a number of business applications which, according to the company, make more than one billion calls to Crunchbase’s API each year.
Pay attention because: As news cycles tighten and anything more than a few hours old becomes old news, readily accessible information on platforms like Crunchbase has grown in importance. Meanwhile, such platforms eliminate hours of manual research for sales reps and investors seeking new opportunities.
The work: Amplitude’s data tools offer detailed insights into customer behaviors. The company says its behavioral analysis platform can automatically identify personalities that are likely to gravitate toward a particular product or service, then map retention rates and paths to maximize them.
Pay attention because: Marketing dollars are precious. By segmenting a company’s user base, Amplitude’s technology shows which types of customers to prioritize. Watch for businesses that use Amplitude’s technology — including Microsoft, Instacart, PayPal and Capital One — to build product experiences, messaging and sales funnels to cater to specific users.