RavenDB is a document database, and unlike a relational database (like say Microsoft SQL Server) – you can’t do your usual group-by type queries quite as simply.

Consider this scenario: I have a series of documents showing when users have performed certain actions within an application.
I want to produce a unique count of users, and a count of Application starts.

public class StatisticEntry
    {
        public DateTime Timestamp { get; set; }
        public string UserId { get; set; }
        public string Action { get; set; }
    }

 

My documents will look something like:

{ "Timestamp": "2010-01-01", "UserId": "Able", "Action": "Open"}
{ "Timestamp": "2010-01-01", "UserId": "Barry", "Action": "Open"}
{ "Timestamp": "2010-01-02", "UserId": "Barry", "Action": "Close"}
{ "Timestamp": "2010-01-03", "UserId": "Barry", "Action": "Open"}
{ "Timestamp": "2010-01-03", "UserId": "Charlie", "Action": "Open"} 

In SQL Server, I would do something like:

SELECT UserID, count(Timestamp) as Count
FROM StatisticEntries
WHERE Action=”Open”
GROUP BY UserID

This would give me:

Row # UserID Count
1 Able 1
2 Barry 2
3 Charlie 1


For RavenDB instead I need to use a Map-Reduce query. The basics of this are that the first part (the Map) runs over the documents, and finds the matches. The reduce then takes the found matches, and summarises (reduces) the result set.

To make this magic happen, we need to know the results we want out at the end, and define this as a class:

public class StatisticResult
{
    public string UserId { get; set; }
    public int Count { get; set; }
}

We then create our index like so, adding a reference to Raven.Abstractions.dll

public class StatisticsUniqueUsers : AbstractIndexCreationTask<StatisticsEntry, StatisticResult>
{
    public StatisticsUniqueVisitors()
    { 

        Map = docs => from doc in docs
                      where doc.Action == "Open"
                      select new
                                 {
                                     UserId = doc.UserId,
                                     Count = 1
                                 };
        Reduce = results => from result in results
                            group result by result.UserId
                            into g
                            select new
                                       {
                                           UserId = g.Key,
                                           Count = g.Sum(x => x.Count)
                                       };
    }
}

You then feed this class to RavenDB using this bit of magic (This took me a while to find):

Raven.Client.Indexes.IndexCreation.CreateIndexes(typeof(StatisticsUniqueUsers).Assembly, store);


NB: This will result in scanning this entire assembly for indexes. All of them will be created at that time. If any are invalid, you’ll get exceptions.

The CreateIndexes method should be called when your application starts – but be aware that you may get stale results temporarily while RavenDB produces the index.

So, Let me break the index class we created above:

The Map gives us a list of matching documents, with the intermediate results being:

{ "UserId": "Able", "Count": 1}
{ "UserId": "Barry", "Count": 1}
{ "UserId": "Barry", "Count": 1}
{ "UserId": "Charlie", "Count": 1}

The Reduce then summarises these values and gives us the summarised values:

{ "UserId": "Able", "Count": 1}
{ "UserId": "Barry", "Count": 2}
{ "UserId": "Charlie", "Count": 1}

…so, we then have this index, which gives us a per-user count. But we just wanted unique users and total Opens?  Quite right.. It’s a little bit of a round trip, but we get there in the end.

We get the results by performing two queries:

int uniqueUsers = _documentSession.Query<StatisticsEntry, StatisticsUniqueUsers>().Count();
int totalOpens = _documentSession.Query<StatisticsEntry>().Where(s=>s.Action=="Open").Count();

The biggest ‘wtf’ for me was how to actually load the indexes into RavenDB – since there’s no apparent functions to do so on the IDocumentStore interface.

Last year a group of folks ordered Pizza from Crust for lunch .

Unfortunately, Crust ended up being an hour or so late with the order. By way of compensation, Crust refunded us half the order cost. I think we got about $75 back. 
Rather than divide the money back up amongst the people who had ordered, we decided to donate it to a charity.

The charity of choice? Oxfam. They had chickens and goats we could donate to a family – a fitting use for the money.
I got the task of ensuring the money got to Oxfam.

Unfortunately, within a week or two the spam started.
First, it was a subscription to their e-Newsletter (which I’m certain I opted out of in the donation process). This I promptly unsubscribed from. 
I started getting calls to ask for donations – after the second or third call within a month I told them never to call again.
Then the paper spam started. Letters, newsletters, drives for specific events.  I looked for a way to unsubscribe, but it only appears to be possible to do so by sending them a fax or letter.

On average, I’d say I get something every fortnight.  But, somehow they screwed up their data entry and so I get two copies of it. Joy!
Last week I received a pair of letters (they went, unopened, into the recycling bin like usual) on Thursday or Friday. 
Tonight I get back from a week in Melbourne to find another pair of letters.  I tore one of them open and checked for an online way of unsubscribing – but no, still nothing.

I snapped, and wrote the below letter to them. Hopefully it gets my point across quite clearly. It’d be even better if they chose to actually take action and stop sending me this crap.

 

To whom it may concern,

When a group of people from my former workplace chose to donate a small sum of money to Oxfam due to a stuff-up of our lunch order, we expected that it would go to actually purchasing the goats and chickens we selected.

What I certainly didn’t expect was that for a year afterwards (as the chosen coordinator of that group) I’d still be receiving letters and newsletters seemingly every second week asking for more money.

What makes me even more annoyed is that somehow, someone screwed up the data entry, and I get TWO copies of the same letter at the same time. Talk about waste.

You seemingly offer no ability to unsubscribe without me needing to mail you back a "cease and desist".

Please forward this email to whomever is in charge of this strategy – it’s obviously outdated, flawed, and diverts money from donations* and results in serious amounts of very real waste being sent to landfill where it can’t be recycled. 
(* I’m assuming this is diverting money from donations here, because the data entry, mailing systems, printing, etc surely costs you something to operate)

While you’re at it – please make sure to update the website to either make it possible to remove yourself from mailings, or if it’s already possible – then make sure it shows up on the site when searched for. Searches for "unsubscribe" and "mailing lists" refer only to the electronic form.

Additionally, please make a modification to your online donation system that allows people to pre-emptively remove themselves from these mailing lists.

Then, finally, please ensure you remove me from all your mailing lists.

The address in question is:
     [redacted]

Please ensure you get BOTH entries while you’re in there.

Sincerely.

0 comments

Progress

Progress, of a sort. I was originally planning to just stick a faster boot drive into the current server (as I believed that was the main performance bottleneck).

Instead, Softlayer had a special on, which meant it was significantly cheaper to get a new box – with more ram, and the faster drive.

  2009 2009-2010 2010-
OS WS 2008 WEB x32 WS 2008 R2 STD x64 WS 2008 R2 STD x64
CPU Athlon 64 X2 4200+ Xeon X3220 Xeon 5140

Speed

2.2Ghz 2.40Ghz 2.33Ghz

CPUs x Cores

1×2 1×4 2×2
RAM (GB) 2 6 12
Disks      

Boot/OS

160GB 7.2kRPM SATA2 250GB  7.2KRPM SATA2 150GB 10KRPM SATA2

Data

n/a   250GB  7.2KRPM SATA2

Local Backup

160GB 7.2kRPM SATA2 250GB  7.2KRPM SATA2 250GB  7.2KRPM SATA2
Network      

Public

100Mbit 100Mbit 100Mbit

Private

n/a 100Mbit 100Mbit
Remote Management Card (IP-KVM) n/a Yes Yes
       
Price (USD) $129 $214 $269

Onwards and upwards.

Migrations for sites and services will be happening before December.

Kiva’s mission is to connect people, through lending, for the sake of alleviating poverty.

Kiva empowers individuals to lend to an entrepreneur across the globe. By combining microfinance with the internet, Kiva is creating a global community of people connected through lending.

 

For several years our family has decided to forgo giving presents to each other for Christmas. Instead, most of us have decided to donate whatever they would have spent on presents to a charity (such as Médecins Sans Frontières, Technical Aid for the Disabled NSW, Guide Dogs Australia, etc). I decided last year to try a different approach and ‘donate’ to Kiva.

Kiva is a microfinance organisation, who provide small loans to people with low income. The basic idea is this: You lend money (a multiple of USD$25) to an entrepreneur, who uses it to start/expand a business. Over the period of the loan, they repay the loan amount to your Kiva account. No interest is charged.

Often times the amounts are relatively small – a few hundred dollars – but these loans help the people in communities where access to loan services are limited or expensive (loan sharks), and most often traditional banks are unwilling or unable to operate.

Of course, because this is a loan (albeit, with no interest charged by Kiva) – there’s a risk that the loan might not be repaid, or only repaid in part.
I went into the experiment fully expecting that I’d never see the money again, but interested in what the results would be.

My first loans were carefully reviewed, and ended up being to women operating small businesses in Asia on pretty short repayment periods (6-12 months). Within a month, I’d already been repaid enough to make another loan, to which I added some more seed capital, and made a few more loans.

Now I’m up to my 43rd loan – and I havn’t added any more capital since about February. The money has gone through many hands – and the total cost to me has been about 15 minutes every month, and less than lunch out once a week.

My only ‘scare’ with non-repayment was last month with a field partner (the organisation that actually coordinates loans and repayments) who seems to have hit some difficulty. But, it seems that Kiva’s monitoring prevented any significant loss, and all the loans were refunded.

Given some of the locations where these loans are going to, a 100% repayment rate is nothing short of outstanding. All in all – it’s a resounding success as far as I’m concerned.

The task this year is to get the family into it as well.

(This was originally posted by me on AuTechHeads on 28th September, 2010, and is preserved here)

This post is in reply to Nathanael Boehm’s UX for System Architects.

System Architecture is in many ways similar to traditional Engineering or Architecture.
In both cases the end result is almost always going to be used by somebody (Even if it’s just the maintenance team).
In both cases the client will have a number of explicit requirements – I want a three bedroom, two story house, I only have $400k to spend. There will also be a huge number of implied or hidden requirements – The external doors need to be on ground level, the roof mustn’t blow off in the first rain storm.

Knowing these requirements lets an Architect produce a design that meets the requirements as best as possible. Sane Architects will include some flexibility into the design – say having stronger joists than absolutely required so another floor can be added later.

Even with the most flexible design, there are some fundamentals that can’t be easily changed: These are our foundations, and the limitations which we will need to work within. Hosting a premier league football match in the back yard is clearly outside the design requirements for your average suburban house.

Technology continues to improve at a stunning pace. Some things which were almost impossible are now done with a little effort. Other things which were once difficult are now easy.

The Architects can often do a better job explaining what these decisions are, and what limitations this will place on the system.
Clearly UX and the System Architect folks need to talk more.

0 comments

Leaving Massive

For the last three years (give or take a month or so) I’ve been working with the great folks at Massive.
Time has come for me to move on and find other great opportunities.

If you know of any interesting .NET Jobs opening up in Sydney (or Melbourne), please let me know.
I’ll be available to start from about the 16th of November. Update: I’m no longer looking for work, thanks.

Things I’m currently interested in:

  • Playing with cool gadgets: Android & Windows Phone 7
  • Multithreading & Parallel Processing in .NET 3.5 and 4.0. (ParallelFx, Rx, etc)
  • IPTV / Streaming Media Delivery
  • ASP.NET MVC

Things I’ve been working with most recently:

  • ASP.NET Web Apps (.NET 3.5/WebForms)
  • RESTful WCF Services
  • Building Custom Serialisation/Deserialistion Adaptors & APIs for third party services.
  • Build Management

To all the folks at Massive: It’s been a great ride; Working with some of the smartest folks I know. Thanks for all the fish.

(Originally Posted by me on AuTechHeads, April 27th, 2010, and preserved here)

Why Cloud Computing Services have huge stumbling blocks to their adoption for the projects I work on.

One of the projects I’m working on has a need to switch to an service bus / message queue system.

We’re after something that’s fairly light-weight. Ideally something we can package into our existing distribution and manage the configuration as part of our existing application’s configuration.
We also need some level of reliability – we’re not expecting clients to go yanking servers out of the rack, but if we send a message, we want to know that it’s going to be delivered.

A few people have suggested Cloud based queue systems as a potential solution. Amazon SQS, Azure AppFabric, and Linxter have all been mentioned a few times.

Unfortunately, no cloud solution is going to pass even a preliminary inspection.

When you use a cloud based architecture, you get to offload some of the responsibility of ensuring the solution is up and running. But at the same time you take a big dependency on the security and stability of not just that cloud provider’s infrastructure – but the entire route from your client to the provider.

If some script kiddie decides that today’s the day he’s going to packet Facebook, and your cloud provider happens to be on the wrong side of a congested router, well – your solution better work fine without it’s message queue.

Any time you add another provider, it complicates the solution. Even if the configuration of the service is as simple as adding a DLL, and two lines of code.

The complexity comes not from configuration files or deployment instructions – but in numerous other peripheral things. Just getting someone to hand over a corporate credit card to pay for the damn thing, even if it IS only pocket change is in and of itself a huge ordeal.  If it’s an ongoing cost, then it’s often nigh on impossible.
Lets not even get into the things you need to do to get firewall ports opened to be able to communicate out, the need for complying with the EU Data Protection Directive, or any concerns a client has about what data is passed over the internet.

Unless there’s a particularly good fit for a client, I doubt we’ll be looking much at cloud based services in the near future.

(Originally posted by me on AuTechHeads, April 26th 2010, and preserved here)

Why a fancy resume is useless if you have no enthusiasm for technology.

One of the things rarely discussed in guides on how to get a job in IT is enthusiasm for technology. I am of the opinion that first and foremost, you need to be a technology geek if you want to work in IT.

Don’t confuse being a technology geek with being the stereotypical pimply-faced, pale skinned, greasy haired dweeb. I talk simply of people who have an innate understanding of some area of IT. The kind of person that hears about some new thing and gets a little (or lot) excited.

Personally, I can’t understand why would anyone choose a job in IT if they didn’t like tech.
It would be like me choosing a career in marketing or interior design. Areas in which I really have zero interest.

Yet when interviewing candidates for software development positions, I find far too many of these people. They tend to express no particular interest in any part of software development in particular, or technology in general.
My only conclusion is that they are in IT because it’s a reasonably well paying job.

The not-so-funny thing about it is that these people have fantastic resumes, with years of experience across all sorts of tehnologies. Often they’ve worked for big name companies.

Perhaps these people work well in development teams where there’s a large corporate structure that treats developers like cogs. But for smaller development shops – you really need your wits about you.
Situations inevitably arise where nobody on the team has experience with some particular technology.

A technology geek will be able to solve the problem (either on their own, or with the assistance of the team) and draw on their own experiences gained while reading about or experimenting with other related technology.

That, in my opinion is why you must be a technology geek to work in IT.

So, readers: What is your opinion – is there a role for non-tech-geek folks in IT?

Atlassian already have some documentation on how to integrate IIS and JIRA.

Unfortunately it requires installing some ISAPI components, and a whole lot of fiddling around.

I wanted to see if I could get Application Request Routing to do the same job. Turns out, yes, you can – here’s how.

1. Make sure JIRA is installed and working on your server.

Let’s say that it’s at http://example.com:8080/

I want to access JIRA via: http://jira.example.com/ – but IIS7 is already using port 80 on that server.

2. Alter your conf/server.xml file in JIRA.

Find the /Server/Service/Connector element, and add two attributes:

proxyName=”jira.example.com”

proxyPort=”80″

The Connector element should now look something like

 <Connector port="8080" enableLookups="false" proxyName="jira.example.com" proxyPort="80">

3. Restart the JIRA Service.

4. Install, if you havn’t already, Application Request Routing 2.0, along with URL Rewriting 2.0

5. Enable Proxying on ARR:

  • From the IIS7 Console, click on {ServerName}.
  • Open Application Request Routing.
  • From the Actions pane on the right hand side, Select  ‘Server Proxy Settings’
  • Check ‘Enable Proxy’
  • Set HTTP Version to ‘HTTP/1.1′

6. Add a new site ‘jira.example.com’, with bindings for http://jira.example.com

7. Add a new URL Rewrite Rule for jira.example.com

  • From the IIS7 Console, click on jira.example.com
  • Open URL Rewrite
  • From the Actions pane on the right hand side, select ‘Add Rules’
  • Choose ‘Blank Rule’
  • Set Match Rule to:
  • Requested URL Matches the Pattern
  • Using Regular Expressions
  • Pattern: (.*)
  • Ignore Case: checked
  • Set Action to:
  • Action Type: Rewrite
  • Rewrite URL: http://example.com:8080/{R:1}
  • Append query string: checked
  • Stop processing of subsequent rules: checked

8. Now, with any luck – you should be able to access JIRA via http://jira.example.com  - if not, something isn’t set correctly.

Setting up Fisheye is almost as simple.

Say Fisheye is set up on http://example.com:8060 and I want to access it via http://fisheye.example.com

Repeat steps 4-8 above, substituting ‘fisheye’ for ‘jira’, and then verify you can access fisheye from http://fisheye.example.com

If you’re also doing .NET Development, or have .cs/.aspx/.asmx files in your repository, then you’ll also need to do the following.

Edit the web.config for fisheye.jira.com

Add the following to just before </system.webServer>

<handlers>
 <remove name="WebServiceHandlerFactory-ISAPI-2.0-64" />
 <remove name="WebServiceHandlerFactory-ISAPI-2.0" />
 <remove name="PageHandlerFactory-ISAPI-2.0" />
 <remove name="PageHandlerFactory-ISAPI-2.0-64" />
 <remove name="PageHandlerFactory-Integrated" />
 <remove name="WebServiceHandlerFactory-Integrated" />
 <remove name="SimpleHandlerFactory-ISAPI-2.0-64" />
 <remove name="SimpleHandlerFactory-ISAPI-2.0" />
 <remove name="SimpleHandlerFactory-Integrated" />
 <remove name="CGI-exe" />
 <remove name="ISAPI-dll" />
</handlers>
<staticContent>
 <mimeMap fileExtension=".cs" mimeType="text/plain" />
</staticContent>
<security>
 <requestFiltering>
  <fileExtensions>
   <remove fileExtension=".config" />
   <remove fileExtension=".csproj" />
   <remove fileExtension=".cs" />
   <add fileExtension=".cs" allowed="true" />
   <add fileExtension=".csproj" allowed="true" />
   <add fileExtension=".config" allowed="true" />
  </fileExtensions>
 </requestFiltering>
</security>

If there are any additional filetypes that are in your Fisheye repository that generate 404 errors when navigating, then add them to the fileExtensions section. First as a ‘Remove’, and then an ‘Add’ with allowed=true. You’ll also need to probably add a mimeMap entry too.

Thanks to @OhCrap for the pointers on enabling .cs serving with IIS7.

Got a Kindle? Use it outside the US?  You’re probably better off setting the Kindle Region to the US.

But Wait, you might shout – You’ll pay roaming charges for that!  Yes, that’s true – you’ll pay USD$1.99 for each book downloaded to your kindle over Whispernet while you’re outside the US.

The thing Amazon doesn’t tell you is that the international kindle prices are exactly the same.

Don’t believe me? See for yourself on the Amazon site.

First, set your Kindle Region to (say) Australia.

Kindle Region is Australia

Now, take a look at some book you’re interested in.

Helfort's War Book 2: The Battle of the Hammer Planets. International Kindle Store Price: USD$8.39, includes Whispernet International Delivery

Setting my kindle region to the USA for a moment and refreshing the page shows me the US Kindle Price

Same book, but on the US Kindle Site. Price? USD$6.39.

As you can see, for this book it’s USD$6.39 on the US Kindle Store. Add in USD$1.99 for International Whispernet Delivery, and it comes to USD$8.38  - You make a saving of USD 1c by doing this.

Things are much the same it seems for magazine subscriptions. Although, I’ve only checked the prices for two monthly magazines, the price differences are about the same.

International Kindle Store Price: USD$5.99, with International Whispernet Delivery

US Kindle Store Price: USD$2.99/month

Plus – there is significantly MORE books available on the US Kindle Store than the Australian version.

It’s well worth checking out.

NB: If you have any magazine/newspaper subscriptions, and especially any daily/weekly ones – then read the roaming letter they send you carefully, it includes information about per-item delivery costs.

Edit: Can I use any address in the USA, Or do I need a valid US Billing Address?

Don’t edit your billing address, just your Kindle Region – in the section marked ‘Your Country’. And yes, any address in the USA Works, as long as you have a valid Zip Code.