Author Archives: Matt

Friendly Reminder: Don’t use platform defaults

Posted on August 24, 2018 by Matt

tl;dr Don’t use platform defaults as they may not be the same across all machines, especially when sending data in between them. Microsoft has a big warning section in the Encoding.Default property docs about this. The story A few years … Continue reading →

Posted in Azure, C# | Leave a comment

Get Azure Functions 2 Preview Up And Running On OS X With C# Functions

Posted on April 17, 2018 by Matt

** This is what I encountered as of April 16, 2018 so this may no longer be accurate. Also, note that this is for the preview version. tl;dr Finally after much frustration and a bit of luck, I got something … Continue reading →

Posted in Azure Functions, C# | Leave a comment

Disappearing Azure Data Lake Files?

Posted on March 8, 2018 by Matt

tl;dr If you upload files from the Azure Data Lake Explorer in Visual Studio, it will set an expiration time on the uploaded files. That expiration time looks to be about one week from the upload time. Once that time … Continue reading →

Posted in Azure, Data Lake, U-SQL | Leave a comment

Friendly Reminder: GetHashCode Isn’t an Identifier

Posted on February 21, 2018 by Matt

tl;dr Don’t use GetHashCode to identify an object. Microsoft has a big warning section in the GetHashCode docs about this. The story We store around 70 million unique objects in Azure Table Storage and we access them through an API … Continue reading →

Posted in C# | Leave a comment

U-SQL – Extracting Information From a Column Containing JSON

Posted on January 17, 2018 by Matt

There are plenty of examples on how to parse JSON files using U-SQL, but what if one of the columns of a TSV contains JSON like this example: ID Name Vehicle 1 Joe "{""Year"":""2017"",""Make"":""Ford"",""Model"":""Fusion""}" 2 Bob "{""Year"":""2016"",""Make"":""Kia"",""Model"":""Sorento""}" We can adapt … Continue reading →

Posted in Azure, Big Data, JSON, U-SQL | 2 Comments

Using Azure Functions to Replace Cloud Workers

Posted on December 8, 2017 by Matt

When looking at options for replacing Worker Roles in Azure Cloud Services, more and more, Azure Functions look to fit the bill. I like the idea of just writing code that focuses on what needs to be done and not … Continue reading →

Posted in Azure, Azure Functions | Leave a comment

Repost: Filtering columns out of data using Azure Data Factory

Posted on December 6, 2017 by Matt

At work, every night we ingest a couple of large comma separated files (CSVs) that are 45 to 50 GB each. These files are generated externally, so we don’t control what information is in them. For our purposes, we really … Continue reading →

Posted in Data Factory | Leave a comment

Whoops, starting over (or why you should really keep your contact info up to date)

Posted on December 5, 2017 by Matt

Last year, I started blogging again. Nothing major, just tidbits from things I struggled with at work or found interesting. Back in September, my hosting at dreamhost expired, they sent out warnings and I never got them. It’s not their … Continue reading →

Leave a comment