AbstractPath | AbstractPath http://www.abstractpath.com thoughts from @tumtumtum Sat, 20 Jun 2015 19:57:34 +0000 en-US hourly 1 http://wordpress.org/?v=4.2.2 Automatically deploying TeamCity Azure Build Agents into specific virtual networks (vnets)http://www.abstractpath.com/2015/automatically-deploying-teamcity-azure-build-agents-into-specific-virtual-networks-vnets/ http://www.abstractpath.com/2015/automatically-deploying-teamcity-azure-build-agents-into-specific-virtual-networks-vnets/#comments Sat, 16 May 2015 01:09:58 +0000 http://www.abstractpath.com/?p=1108 Jetbrains has recently added support for using Azure as cloud build agents. It supports start and stopping existing virtual machine instances or start and stopping new virtual machine instances created from an image. Unfortunately the plugin currently doesn’t support creating new virtual machine instances within a specific virtual network (vnet) which can complicate setups where you want to keep all your development machines within a single vnet with all the associated benefits such as easy DNS lookups.

I’ve created an updated version of Jetbrains’ plugin to allow you to specify a virtual network. You can download the plugin directly here or browse the source code on GitHub.



http://www.abstractpath.com/2015/automatically-deploying-teamcity-azure-build-agents-into-specific-virtual-networks-vnets/feed/ 0
Strange Visual Studio 2013 debugger bughttp://www.abstractpath.com/2014/strange-visual-studio-2013-debugger-bug/ http://www.abstractpath.com/2014/strange-visual-studio-2013-debugger-bug/#comments Mon, 20 Oct 2014 21:41:54 +0000 http://www.abstractpath.com/?p=1098 var bar =... line. The code should throw an InvalidOperationException but instead gets to Console.WriteLine().It works fine if you don’t set a breakpoint or just run without the debugger attached.
using System;
namespace ConsoleApplication1
    public class Foo<T>
        public virtual T GetBar<K>(K foo)
            throw new InvalidOperationException();
    class Program
        static void Main(string[] args)
            var foo = new Foo<string>();
            // Place a break point on this line and then step over.
            // An exception should throw but it doesn't.
            // If you try to step into instead it will not go into GetBar()
            // The instruction pointer is buggered.
            var bar = foo.GetBar(new Guid());
            Console.WriteLine("Should not get here");
http://www.abstractpath.com/2014/strange-visual-studio-2013-debugger-bug/feed/ 0
Opening the back cover of a Tesco Nokia 100http://www.abstractpath.com/2014/opening-the-back-cover-of-a-tesco-nokia-100/ http://www.abstractpath.com/2014/opening-the-back-cover-of-a-tesco-nokia-100/#comments Sat, 19 Jul 2014 09:46:10 +0000 http://www.abstractpath.com/?p=1093 I just bought a Tesco Nokia 100 phone (only £9!!) to use for my apartment’s annoying GSM based intercom. It’s got 35 day stand by battery so that makes it a good for the job. Following the instructions for opening the back cover will make you extremely frustrated. Here’s a short video showing the easiest way to open the phone.

Click here to view the embedded video.

http://www.abstractpath.com/2014/opening-the-back-cover-of-a-tesco-nokia-100/feed/ 0
Gaining additional memory when running Linux on Azure Extra Small Instanceshttp://www.abstractpath.com/2014/gaining-additional-memory-when-running-linux-on-azure-extra-small-instances/ http://www.abstractpath.com/2014/gaining-additional-memory-when-running-linux-on-azure-extra-small-instances/#comments Sun, 29 Jun 2014 13:14:15 +0000 http://www.abstractpath.com/?p=1048 If you use Azure to host your Linux VMs (I prefer Azure extra-small instances over EC2 tiny instances because they allow you to put swap on ephemeral storage) you may notice that you only have 541MB of free memory rather than the 768MB promised by Microsoft. This is because the Linux crash kernel by default takes up ~128MB on an extra-small instance. You can change this by removing the crashkernel=auto reference inside your /etc/grub/grub.conf.


[root@localhost ~]$ free -m
             total     used     free     shared    buffers     cached
Mem:         541        306     235         0         12        175
-/+ buffers/cache:      118     422
Swap:         2047        0     2047


[root@localhost ~]$ free -m
             total     used     free     shared    buffers     cached
Mem:         670        299     370         0         12        165
-/+ buffers/cache:      121     548
Swap:         2047        0     2047

http://www.abstractpath.com/2014/gaining-additional-memory-when-running-linux-on-azure-extra-small-instances/feed/ 0
Bloom.fm technology stackhttp://www.abstractpath.com/2014/bloom-fm-technology-stack/ http://www.abstractpath.com/2014/bloom-fm-technology-stack/#comments Fri, 23 May 2014 22:14:32 +0000 http://www.abstractpath.com/?p=977 Bloom logoWith the unfortunate demise of Bloom.fm, a product I designed and developed as CTO for 2.5 years at a company I co-founded 5.5 years ago; I think it’s a good time to write a series on my learnings, decisions and experiences.

My development team (minus visual designers) varied from 6-13 developers over the years. With this size team we wrote a music streaming service backend, a music content ingestion system, 3 native mobile apps, a web app, an internal admin console and designed and built in-house infrastructure for hosting everything. We did not use cloud hosting because of the need to store over 700TB of content securely and without the massive on-going costs that cloud storage would incur. Here’s a quick list of the tech we used at Bloom…

Development tools

I spared no expense with our development tools. Each developer had a fast workstation with dual 27″ monitors. We had about 18 development servers which included simulated production environments (dev & test) and 9 Windows and OSX build servers for continuous integration and testing.

  • TeamCity for continuous integration
    • All builds and tests were run automatically and logged
    • Custom deployment tasks and services were written to allow one push deploy to dev, test and production
  • Confluence for documentation
  • Mercurial for source control
    • Branching was used extensively.
    • Commented out code was not allowed to be checked into the repository.
  • Visual Studio 2012 for backend and web development
  • Xcode for iOS development
  • IntelliJ IDEA for Android development
  • YouTrack for bug and feature tracking, project management
Backend web services

Everything ran on CentOS 6.5 except for the C# web services which ran on Windows 2008 R2. Each web service server (“AppServer”) was stateless with new servers able to be added at any time to increase capacity. Shared transient state (e.g. user session data, unique streaming request tokens, search queries) was stored using Redis. Web Services were accessed by  the mobile and web apps using JSON over HTTP. Every new web service written in C# triggered generation of the appropriate client-side JavaScript, C++, C#, Objective-C and Java code. At our peak we were handling over 10,000 concurrent API requests per second.

Every element of the backend was implemented with care and attention to detail to architecture and performance. For example, our caching strategy was to cache structured data (rather than blind output caching) which allowed caching of combined static (e.g. search results) and dynamic (e.g. user purchase state) data. We used a Redis pub/sub to support distributed cache invalidation which allowed us to use AppServer memory for local caching of shared data that would otherwise need to always be accessed via Redis.

Because we started a long time before tech like SignalR was released, we developed our own scalable real-time notification service based upon http.sys and Memcached (later Redis) back in 2008. This notification service allowed us to deliver hundreds of thousands of notifications without scalability issues. Users could edit a playlist on one device and see it change immediately on their other device.

(team size: 7 people including myself)

  • C# 4.5, WCF and IIS7.5 for web services
  • Postgres 9.4
  • Sqlite for simulated fast unit testing
  • Redis for structured caching (search results, user, artist, album data, personalised radios etc) and distributed state (session state, streaming tokens, etc)
  • Solr for search (we originally used NLucene)
  • RabbitMQ for queueing and request offloading
  • NUnit for unit testing
  • Booksleeve redis client
  • Shaolinq Linq provider for Postgres and Sqlite
  • Platform.VirtualFileSystem for file system access
  • StructureMap for dependency injection

There were 3 VLANS on the network. Management, Internet and Backend. The backend VLANs all ran on 10Gb ethernet. All of Bloom’s infrastructure was managed in-house at co-location facilities in central London and docklands. Bloom ran on 60 servers (approximately 40 physical servers and 20 virtual servers). All physical servers were Dell PowerEdge servers (R620s, R720s). Storage servers consisted of 14 clustered 4U SuperMicro storage servers running our own setup of CentOS and Gluster.

The total cost of infrastructure was a little over £200K (capex). I worked my ass off to make the most efficient custom built solution — and it was awesome fast.  This one off cost was less than the ongoing monthly spend our CEO allocated to banner ads with an insane CPC of over £200 per user. Cloud hosting is great for many startup business models where costs scale directly with user count (like Dropbox) but if we had hosted Bloom.fm on Amazon it would have cost well over £200K every month (not just a one off capex!).

(team size: 2 people including myself)

  • Linux LVS for load balancing
  • Zabbix for monitoring
  • Puppet for automation and server management
    • EVERYTHING was scripted. New servers were automatically setup in a just a few steps
  • Graphite for API logging and monitoring
  • NGINX for serving images and media content
    • PHP & PHP-FPM to manage streaming authentication via Redis (time limited tokens etc)
    • Validated requests were redirected back to NGINX using X-ACCEL-REDIRECT
  • GlusterFS for storage (two 800TB volumes storing over 400 million files each)
    • Each brick was used as an independent server and made failsafe using LVS allowing mounts to work right down to the failure of the last brick in the cluster
    • Each brick was formatted using xfs with 512 byte inodes
    • Insane performance which increased every time new storage was added
    • One-off cost was £140K. Would have cost over £70K a month if we had stored in the cloud (not even including data transfer costs!)
  • XenServer as the hypervisor for our VMs
  • FusionIO PCI flash storage for primary databases (excellent value)
  • Force10 10Gb switches (excellent performance and value)
  • Cisco ASR 1000 routers
  • Pingdom for SMS monitoring
Mobile apps

All mobile apps were written natively for performance and size.

(team size: 4 people including myself)

  • Objective-C for iOS
  • Java for Android
    • The UI used OpenGL for animation performance (rare for any app outside of games)
  • C# for Windows Phone
Web Apps

We had two web apps. The Bloom.fm music player web app and the internal admin system for managing and monitoring the system as well as user management.

(team size: 2 people)

  • C# 4.5
  • MVC
  • Razor
  • JQuery
Third Party Integration

Bloom integrated with many third party services including:


We built an extremely comprehensive music streaming service with a comprehensive feature set that easily handled 100,000 concurrent users with almost no load and could have handled ten times that without any additional servers or optimisations. This was all done (including computers, servers, salaries) on a technical and (graphical) design budget of £3M over 2 and 1/2 years. Many companies can easily spend upwards of £1M on storage alone (don’t use an off the shelf SAN for a music streaming service!). The key was to keep all design and development in-house (contractors are ok as long as they work in-house), hire only the best of the best and to be confident and capable enough to develop custom solutions when necessary. A tech startup creates tech right?

http://www.abstractpath.com/2014/bloom-fm-technology-stack/feed/ 1
Updating OpenSSL to Install Postgres 9.x on CentOS 6.xhttp://www.abstractpath.com/2014/updating-openssl-to-install-postgres-9-x-on-centos-6-x/ http://www.abstractpath.com/2014/updating-openssl-to-install-postgres-9-x-on-centos-6-x/#comments Sun, 12 Jan 2014 01:10:43 +0000 http://www.abstractpath.com/?p=937 The instructions provided here for installing Postgres on CentOS via yum don’t work for Postgres 9.x (Postgres 9.3 being the latest) if you’re stuck running CentOS 6.4 or lower. CentOS 6.4 and lower do not ship with OpenSSL 1.0.1 which is required by Postgres 9.x — they ship with OpenSSL 1.0.0 or lower.

You will most likely get the following errors:

 Error: Package: postgresql93-Server-9.3.2-1PGDG.rhel6.x86_64 (pgdg93) Requires: libssl.so.10 (libssl.so.10) (64bit)</p>
<p>Error: Package: postgresql93 -libs-9.3.2-1PGDG.rhel6.x86_64 (pgdg93) Requires: libssl.so.10 (libssl.so.10) (64bit)</p>
<p>Error: Package: postgresql93-9.3.2-1PGDG.rhel6.x86_64 (pgdg93) requires: libssl.so.10 (libssl.so.10) (64bit)</p>
<p>Error: Package: postgresql93-libs-9.3.2-1PGDG.rhel6.x86_64 (pgdg93) Requires: libcrypto.so.10 (libcrypto.so.10 ) (64bit)</p>
<p>Error: Package: postgresql93-Server-9.3.2-1PGDG.rhel6.x86_64 (pgdg93) Requires: libcrypto.so.10 (libcrypto.so.10) (64bit)

Follow the instructions here to build and install an RPM package for OpenSSL 1.0.1. Once that’s done, follow the instructions for installing Postgres again and it should work.


An even simpler approach

Install the ius repo:

 rpm -ivh http://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/ius-release-1.0-11.ius.el6.noarch.rpm

Enable the ius-archive repo:

Edit /etc/yum.repos.d/ius-archive.repo and enable the ius-archive repo by changing enabled=0 to enabled=1.

Install the yum replace plugin:

yum install yum-plugin-replace

Replace the existing openssl package with openssl10:

yum replace openssl --replace-with openssl10

http://www.abstractpath.com/2014/updating-openssl-to-install-postgres-9-x-on-centos-6-x/feed/ 0
C# LINQ and Sum, Min, Max, Average behaviourhttp://www.abstractpath.com/2014/c-linq-and-sum-min-max-average-behaviour/ http://www.abstractpath.com/2014/c-linq-and-sum-min-max-average-behaviour/#comments Fri, 03 Jan 2014 22:38:01 +0000 http://www.abstractpath.com/?p=888 I’ve recently been improving support for aggregates in Shaolinq in order to make Shaolinq queries match LINQ to objects as closely as possible. The LINQ Enumerable<T>.Sum, Enumerable<T>.Min, Enumerable<T>.Max and Enumerable<T>.Average extension methods map to the SQL aggregate functions SUM, MIN, MAX and AVG respectively. In SQL, all these functions will return a null if there are no rows returned. In .NET, the behaviour when the enumerable is empty depends on whether the type T is nullable or not.

The .NET aggregate methods Min, Max and Average will throw an exception if the enumerable is empty and type T is not nullable (int vs int? etc). If type T is nullable then Min, Max, Average will return null. The exception to all this is Sum which will always return 0 regardless of the nullability of type T.

When performing a SQL aggregate query such as SELECT SUM(age) FROM PERSON, SQL will return a single row even if there are no PERSON rows and the SUM column value set to null.  This null value is used to detect empty result sets which can be mapped to the appropriate exception or null result depending on the nullability of the .NET required type. To match the behaviour of always returning 0 for Enumerable<T>.Sum regardless of whether the resultset is empty or not, Shaolinq wraps SUM aggregate calls in a SQL COALESCE. This is handled by the SumAggregatesDefaultValueCoalescer class along with SqlQueryProvider and ObjectProjector.


http://www.abstractpath.com/2014/c-linq-and-sum-min-max-average-behaviour/feed/ 0
Art Tech AT-6 Texan Replacement Retractshttp://www.abstractpath.com/2013/art-tech-at-6-texan-replacement-retracts/ http://www.abstractpath.com/2013/art-tech-at-6-texan-replacement-retracts/#comments Fri, 27 Sep 2013 14:52:43 +0000 http://www.abstractpath.com/?p=879 I have a Art Tech AT-6 Texan RC plane. The one big problem I’ve had with this plane is that the electronic retracts that come with it are prone to overloading, hanging and overheating. If you switch the retracts on/off too fast it causes a current surge which overloads the control transistor causing the retract to hang until it’s unplugged. Sometimes the “brake” switch at either end of the retract’s extents stop working causing the retract motor to continuously push, damaging the “brake” switch even further and causing the whole unit to massively overheat.

A set of replacement retracts for the AT-6 cost £35 but you can get the exact same retract system (sans the wheel and strut which you can move over from the broken set) for £5 each from HobbyKing. They have UK stock as well….

Art Tech AT-6 Texan RC Plane Replacement Retract Replacement retracts with original wheel and strut attached


http://www.abstractpath.com/2013/art-tech-at-6-texan-replacement-retracts/feed/ 0
iPhone exchange setup connection to server failed errorhttp://www.abstractpath.com/2013/iphone-exchange-setup-connection-to-server-failed-error/ http://www.abstractpath.com/2013/iphone-exchange-setup-connection-to-server-failed-error/#comments Wed, 26 Jun 2013 12:03:00 +0000 http://www.abstractpath.com/?p=865 I got a replacement iPhone 5 today and tried to setup my exchange account. It failed and all I got was a cryptic ‘connection to server failed’ error alert box. This error could mean a lot of things and I had no luck finding anything of substance in the Exchange 2010 or IIS logs.  

After a few hours of frustration an searching, I discovered that my problem was caused by having too many mobile devices attached to my exchange account (apparently I’ve owned 10 unique iPhone/iPads over the last 5 years). You can check this by using the exchange web mail interface. Log into web mail -> Options -> All Options then you’ll see a tab on the left named “Phone”. From there you will see a list of devices that have ever synced with your exchange account. Remove the old ones. You may have to delete and then recreate the exchange account on your iPhone for things to start working again.

Exchange Mobile Devices

http://www.abstractpath.com/2013/iphone-exchange-setup-connection-to-server-failed-error/feed/ 0
Tips for running a tech startuphttp://www.abstractpath.com/2013/tips-for-running-a-running-a-tech-startup/ http://www.abstractpath.com/2013/tips-for-running-a-running-a-tech-startup/#comments Sat, 18 May 2013 21:39:24 +0000 http://www.abstractpath.com/?p=824
  • Do not forget that your most important task is to focus on creating a great product. No amount of marketing spend will make up for a crap product that no one really wants to use. 
  • Make what you know — not what you think will make money. Running focus groups to “discover what is needed” is something best left to large corporations with money to burn and little or no vision or decision making capabilities remaining. Your chances of success  are greatly increased if you create your startup to answer a need that you yourself have. In doing this this, you will not have to constantly ask if you are doing the right thing. You also guarantee that you will have at least one user: yourself.
  • Choose one or two things you want to do really well and then set out to do them really well. Do not allow yourself or those who work for you to lose to lose identity and focus. Do not create a homer.
  • Do not forget to create a strong company culture.  Do not expect it to create it itself — especially if you are the founder. Think about what your company’s goals are and then think about the people you will require to accomplish those goals. For example, if your product is a software product then do not deny that you are a technology company. If you’re a technology company then create a culture that will attract technology people. Engineers like casual dress, free coffee, free soda, free snacks, fast computers, big monitors, comfy chairs and a quiet working environment. Providing these things will be one of your most cost effective spends and your investors will thank you for it.
  • Do not forget to create a strong brand. A brand is not just about a logo and colours. A brand projects your company’s culture and values. Like your company culture, the brand will not create itself. The brand needs to be lucid and unified. Don’t forget that the brand is for everyone; most especially your employees. A strong brand will reduce conflict in decision making. (Think of the question: “Is this constitutional?”).
  • Do not focus on small problems whilst ignoring the large problems. For example, do not stress over the cost of a coffee machine for your developers whilst ignoring overrunning third-party consultancy costs (such as SEO consultants).
  • Do not hire SEO consultants.
  • Do not outsource your core competency. If you’re a game company, you can outsource your web design but don’t outsource your game design and development!
  • Avoid anyone who regularly uses the terms viral or game changing as if it’s a solution.
  • Avoid anyone who thinks that the key to success is the silver bullet of the week (for example: changing the color of your buy button from grey to blue is not creative thinking nor will it magically double your user count).
  • Look for the best engineers and pay them accordingly. Simple.
  • Except for sensitive financial information, do not create a cloak and dagger culture. If you’re a tech startup, your employees will (if you’re doing your job properly) be above average intelligence. Respect that.
  • If you’re an engineer yourself, you probably can’t wait to get your hands dirty in the code.  Encourage rather than discourage your team to code. Managers at Google and Facebook are encouraged to code. Software engineers like to follow managers who can code just like soldiers like following orders of generals who can fire a gun.
  • Software scales very well economically. Do not skimp on the initial development costs. Invest upfront to create a great product because the cost of distributing a great product to 10 million users is going to cost about the same as distributing it to 10 users. If you don’t create a great product, you won’t have 10 million users. Someone else will.
  • In the physical world, a premium product has to have a premium price tag to cover the costs of materials and production. In the software world, a premium product does NOT necessarily have to have a premium price tag. If you try to compete purely on price (rather than on quality or execution) then someone else will get your users.
  • Limit your product functionality for up-sell if required but don’t think that deliberately creating a poor user experience in the hopes that people will upgrade is a good strategy. What is most likely to happen is that the user will not pay at all and use an competing product instead. A good example is coffee shops. I once heard it argued that coffee shops should have uncomfortable chairs to encourage faster throughput of paying customers. This sounds reasonable but it does not take into account competition. You may be able to get away with this if you are the only coffee shop in town but in reality your customers will quickly find better, more comfortable places to drink and socialise. You may get customers once but they won’t come back. People don’t put up with bad user experiences unless they have to. If you’re building software, think about creating the equivalent of comfy chairs for your users. As mentioned in the previous point, the amortised cost is minimal. Frustratingly, my co-founder believed in making chairs as uncomfortable as possible and this was the cause of disagreement for some time.
  • At some point you will need some marketing but don’t idiotically spend all your money on marketing. An advertising blitzkrieg will not save you from failure.
  • Have fun :-)
  • ]]>
    http://www.abstractpath.com/2013/tips-for-running-a-running-a-tech-startup/feed/ 1