0 comments

Streaming Twitter API

I’ve been playing with Twitter’s new Streaming API over the last few days.

The old XMPP protocol provided your traditional ‘Following’ stream, plus keyword searching with ‘track’.  All of it being delivered in near-real-time (i.e, immediately after the Tweets were sent).

The new Streaming API uses Comet to deliver a similar set of functionality.

I’ve created a prototype client that uses these functions, so that we can experiment with the new API, the plan being to bring that into MahTweets at some point.

There is some interesting behaviour with the API, particularly with regards to following people.  There was a change to the Web API some months ago, where Twitter disabled the ability to see all tweets your friends make. Basicly they set it so that you can ONLY see the Tweets your friends make, if they’re either Open (not replies), or Replies to someone else you follow.

An example might be that you follow Amy and Barry. It might not be such a big deal for you that you don’t see their tweets to Christine  – but if you, Amy and Barry were discussing something – replies to Christine (even in the same conversation) won’t be seen by you.

Well, the new Streaming API changes that – not only does it bring back that old @replies functionality, it also adds an interesting twist.

Under the Streaming API you also see public @replies to anyone you follow. That is: If Christine were replying to Amy or Bob – you’d see those tweets, even though you don’t follow Christine.

There are some restrictions with the API – you need a reasonably fast and stable connection (Dialup, Satellite, or dodgey wireless connections need not apply), you can’t have more than one streaming connection open per account, and you can’t rapidly change the stream settings.  You also need to explicitly define what keywords and user IDs you want to recieve updates for. That’s currently limited to following 400 users, and/or tracking 200 keywords.

Still… it’s a bunch of fun, and can lead to some interesting scenarios.

Here’s a screencap from the demo client. I’m tracking Aeoth’s UID here, but also getting replies to @aeoth

Test Client - Full Follow Stream

Test Client - Full Follow Stream

A few months ago for some unknown reason the Output pane in Visual Studio stopped displaying output from my application.

I’d get the build notices, exceptions, and thread/process exit information but any calls to Debug or Console to output information wouldn’t display.

tickprogramoutputIt turns out that you can de-select “Program Output” – and somehow it’d become deselected. Even now it still turns itself off, apparently by random.

Right clicking in the Output pane should let you re-select that value.

Dear New York Times,

Whilst I understand you’ve been around some considerable time, and that you do have journalists to fire, you do need to know your nag screen is incredibly annoying. It’s probably also not helping with your ad impressions either.

nytfail

Sure, it might only take a minute to sign up – but it’s really not worth it for me. It’s actually quicker just do add your entire domain to my DNS Block List.

image

That way I don’t have to see your annoying sign-in page ever again.

Have fun with that print media business. I hear it’s a raging success. Oh, wait.

Sincerely,

Will

  • Are you new to Social Media?
  • Are you trying to recruit folks in IT, or related industries?
  • Do you think Twitter might be a great way to get more leads to fill roles?

If you answered yes to the above, then you need to pause for a moment, because Twitter is really not what you think it is.

Think of Twitter more like a 24-7 party where everyone mixes and mingles and interacts however they want – you suddenly turning up and jumping on the Mic demanding attention to pitch people on some “Exciting new Role” is just going to piss people off.

If you’re going to engage with folks, make it PERSONAL – They’re not on Twitter so you can sell them a product, they’re there to interact with other folks.

Spend five minutes or so looking at their profile, their previous tweets, and sites they link to and/or personal site. Interact with them on things that you might have a common interest on, and after you’ve got some sort of relationship (This takes more than 15 minutes) – it might be appropriate to send them a private message about the role.

If they’re not following you, you won’t be able to send them a private message, so that’s probably a good indication they’re Just Not Interested.

Yes, I’ve blogged about this before – but I’m writing it again, because it’s still relevant.

5 comments

Star Trek XI

I got home not too long ago from seeing the Star Trek reboot film by JJ Abrams.

SPOILER WARNING: If you have not seen the film, and you have any intention of doing so – do not continue reading.  The no-spoilers review is: It’s good, go see it.

Read the rest of this entry »

Edit: This post was linked from Stack Overflow – you might want to check back there for more/better discussion about the topic.

Recently Jeff Attwood wrote about how they are using ELMAH to get more information about the types of errors occurring in Stack Overflow.

Effectively ELMAH is designed as a ‘drop in’ fault capturing system for ASP.NET. It works really well there, and for many situations you can get along just fine without even needing to recompile your application (it does need some editing of the web.config though).

I wanted a way to capture more detail about the faults occurring in our dev and production environments, especially when working with WCF – since a lot of error detail tends to be hidden, or is difficult to reproduce.

Dropping in ELMAH into a WCF application will by default mean you miss the vast majority of errors – WCF swallows the error, and doesn’t let it get back up to ASP.NET.

There’s two ways you can go about fixing this:
Side Note: If you’re not hosting WCF in ASP.NET, then Option 2 may not be directly possible for you without some modification.

#1 – Wrap everything in try/catch blocks (if you didn’t already) and sprinkle this line around everywhere:

Elmah.ErrorSignal.FromCurrentContext().Raise(YourExceptionHere);

#2 Add a HttpHandler, and Decorate your Service(s) with an Error Handling attribute.

I borrowed the ServiceErrorBehaviourAttribute code from somewhere else, and I can’t find the source of it at the moment. Effectively this was so I could manipulate the HTTP Status Codes going back to the client when there was an error. It just so happens that this is a great way of capturing Exceptions and sending them to ELMAH at the same time.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Channels;
using System.ServiceModel.Description;
using System.Collections.ObjectModel;
using System.Net;
using System.Web;
using System.IO;
using Elmah;
namespace YourApplication
{
	/// <summary>
	/// Your handler to actually tell ELMAH about the problem.
	/// </summary>
    public class HttpErrorHandler : IErrorHandler
    {
        public bool HandleError(Exception error)
        {
            return false;
        }

        public void ProvideFault(Exception error, MessageVersion version, ref Message fault)
        {
            if (error != null ) // Notify ELMAH of the exception.
            {
                Elmah.ErrorSignal.FromCurrentContext().Raise(error);
            }
        }
    }
	/// <summary>
	/// So we can decorate Services with the [ServiceErrorBehaviour(typeof(HttpErrorHandler))]
	/// ...and errors reported to ELMAH
	/// </summary>
	public class ServiceErrorBehaviourAttribute : Attribute, IServiceBehavior
    {
        Type errorHandlerType;

        public ServiceErrorBehaviourAttribute(Type errorHandlerType)
        {
            this.errorHandlerType = errorHandlerType;
        }

        public void Validate(ServiceDescription description, ServiceHostBase serviceHostBase)
        {
        }

        public void AddBindingParameters(ServiceDescription description, ServiceHostBase serviceHostBase, Collection endpoints, BindingParameterCollection parameters)
        {
        }

        public void ApplyDispatchBehavior(ServiceDescription description, ServiceHostBase serviceHostBase)
        {
            IErrorHandler errorHandler;
            errorHandler = (IErrorHandler)Activator.CreateInstance(errorHandlerType);
            foreach (ChannelDispatcherBase channelDispatcherBase in serviceHostBase.ChannelDispatchers)
            {
                ChannelDispatcher channelDispatcher = channelDispatcherBase as ChannelDispatcher;
                channelDispatcher.ErrorHandlers.Add(errorHandler);
            }
        }
    }
}

Once you’ve added that, then it’s just a matter of decorating your Service like so:

    [ServiceContract(Namespace = "http://example.com/api/v1.0/")]
    [ServiceErrorBehaviour(typeof(HttpErrorHandler))]
    public class MyServiceService
    {
      // ...
    }

…and then making sure ELMAH is added as a reference, and adding it’s entries to your web.config.

Then you’ll be getting a whole stack of errors you otherwise may not have seen.

It’s also possible to log exceptions from higher up the chain (eg Databases, Files, etc) by using the line of code from Option 1.

Issues

Whilst ELMAH is great for capturing information about the request, I havn’t yet found any way to capture the original HTTP Request – this would be the ultimate goal for me.

It’s also not particularly easy to capture additional information (such as database records, or objects in cache, etc) without rolling your own copy of ELMAH.

All in all though – for a few minutes work, it’s one additional way to capture errors that your existing code may not be able to.

Yes, ELMAH even captures errors (in most situations) when your WCF services can’t start up (eg your fubar’ed some attributes).

Hope that helps.

NB: Use this code at your own risk, don’t blame me if it brings down your multi-million-dollar-per-hour application and causes you to go bankrupt.

Caprica-1

If, like me, you decided to go installing stuff off the Web Platform Installer 2.0 Beta, you may start getting IIS7 Application Pool lockups, and general non-responsive behaviour.

Digging through the Event Log, I was getting this:

Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0×47919413, faulting module ntdll.dll, version 6.0.6001.18000, time stamp 0x4791a7a6, exception code 0xc0000374, fault offset 0x000b015d, process id 0xad0, application start time 0x01c9c03e217b6255.

Unfortunately, this doesn’t really give any useful information, and the search results seemed to indicate that it was PHP at fault. I was running PHP, but this happened even when PHP was disabled, and the request was for static files.  PHP was also running just fine under other Application Pools.

After much faffing about, including creating new application pools, even pulling out DebugDiag and getting memory dumps, I was still lost – so I started disabling IIS Modules as a last resort.

I didn’t have to go too far: Removing DynamicIPRestrictionModule caused everything to start working again.

Ironically, this module is supposed to prevent Denial of Service attacks.

Tried reporting this as a bug in Connect, but it doesn’t seem to be supported via there – only via the Security forums on the IIS Site. Keep an eye on this thread to see if there’s a resolution.

Update: Microsoft responded to me, and confirmed it’s a bug in the application, it’s queued for fixing in a later release.

I was away over the Easter long weekend and didn’t get to see the whole ‘amazon fail’ thing happen. Actually, I didn’t even really find out about it until today.

The story goes that sometime on Saturday Amazon suddenly stopped returning pro-lesbian/gay/bi/transgender (LGBT) material in search results – in some cases returning anti-LGBT material instead.

I’ll leave the morality of why this was a bad thing alone – other far more eloquent folks have written about it already – and instead offer my view of just how this can happen.

Some people have commented that they simply cannot believe this was a mistake, and that it must have been a deliberate act by someone at Amazon. Others have said that they cannot believe there isn’t more checks and balances against this sort of thing happening.

I don’t have any special knowledge of the specifics of this situation, but I still believe that it’s within the realm of possibility that this could have been a completely unintentional side-effect of another change.

Here’s a hypothetical situation which could explain what happened.

Let’s suppose that someone doing database administration on the product management side of things for Amazon has the ability to make direct, or semi-direct changes to the data in the database. 

Let’s also suppose that in order to prevent human error, the system that Amazon Employees use has a delay of (say) one hour before changes are sent to the actual production database.  That continual one-hour buffer would have some sort of checks to see if there were unusual behaviour – such as updates to more than some reasonable number of records at any one time.

It’s not unlikely that Amazon have a rather complex categorisation system that allows products to be placed in any number of categories, and that categories can belong to other categories. 

From here it’s but a hop-skip-and-blunder into someone updating a series of categories to ensure they’re marked as adult. A command to mark any category with ‘lesbian’ and ‘sex’ in the title as Adult might seem fairly reasonable if you aren’t careful.  If this is one small series of category updates, it may not trigger any alerts, even though tends of of products are now categorised as Adult by association to those categories. (See Data Normalization Side note below)

Whenever a human is involved in something, there is always a chance for something to go wrong in entirely unexpected ways. The sign of intelligence though is learning from your mistakes and ensuring you do what you can to prevent it from re-occurring.

In Amazon’s case, this might be ensuring that they add a check to see if there is a major difference in the number of Adult products. But no matter how many checks and reviews you have – People will still manage to break things in new and creative ways.

Further Reading:

Side  note: Data Normalization

Because of the nature of computer databases, it’s encouraged (and very efficient) to group common information together, and reduce duplication of data by keeping only a reference to related information.

An example of this might be in a genealogy system – for any particular person, you have information about who the parents are. Instead of storing the Firstname, Lastname, and Date of Birth of each parent – I would instead store the unique identifier for each of the parent’s record.  This act of storing just the unique identifier is part of Normalisation of data. 

When it comes time to display that information about a person’s parents – you look up the record for each parent and retrieve any information needed then. 
This means you also have only one location to update information about any one person – for instance if a person died, you could put the location and date of death on that person’s record. When I then needed information about people who have living parents – I can cross-reference those tables.

Two methods that I keep finding myself needing are a way to Serialize and Deserialize objects in .NET 3.5.

Either for Unit Testing against a WebService of some kind, or for storing objects in memory to disk in XML for human-readable niceties.

Don’t forget to add in appropriate error handling code as needed.

With .NET 3.5 SP1, these methods will serialize (almost) any object to either XML or JSON, it was based in part off an example given in a long since forgotten forum post.

 

using System.IO;
using System.Runtime.Serialization; // System.Runtime.Serialization.dll (.NET 3.0)
using System.Runtime.Serialization.Json; // System.ServiceModel.Web.dll (.NET 3.5)
using System.Text;
namespace Serialization
{
    public static class Helpers
    {
        /// <summary>
        /// Declare the Serializer Type you want to use.
        /// </summary>
        public enum SerializerType
        {
            Xml, // Use DataContractSerializer
            Json // Use DataContractJsonSerializer
        }

        public static T Deserialize<T>(string SerializedString, SerializerType UseSerializer)
        {
            // Get a Stream representation of the string.
            using (Stream s = new MemoryStream(UTF8Encoding.UTF8.GetBytes(SerializedString)))
            {
                T item;
                switch (UseSerializer)
                {
                    case SerializerType.Json:
                        // Declare Serializer with the Type we're dealing with.
                        var serJson = new DataContractJsonSerializer(typeof(T));
                        // Read(Deserialize) with Serializer and cast
                        item = (T)serJson.ReadObject(s);
                        break;
                    case SerializerType.Xml:
                    default:
                        var serXml = new DataContractSerializer(typeof(T));
                        item = (T)serXml.ReadObject(s);
                        break;
                }
                return item;
            }
        }

        public static string Serialize<T>(T ObjectToSerialize, SerializerType UseSerializer)
        {
            using (MemoryStream serialiserStream = new MemoryStream())
            {
                string serialisedString = null;
                switch (UseSerializer)
                {
                    case SerializerType.Json:
                        // init the Serializer with the Type to Serialize
                        DataContractJsonSerializer serJson = new DataContractJsonSerializer(typeof(T));
                        // The serializer fills the Stream with the Object's Serialized Representation.
                        serJson.WriteObject(serialiserStream, ObjectToSerialize);
                        break;
                    case SerializerType.Xml:
                    default:
                        DataContractSerializer serXml = new DataContractSerializer(typeof(T));
                        serXml.WriteObject(serialiserStream, ObjectToSerialize);
                        break;
                }
                // Rewind the stream to the start so we can now read it.
                serialiserStream.Position = 0;
                using (StreamReader sr = new StreamReader(serialiserStream))
                {
                    // Use the StreamReader to get the serialized text out
                    serialisedString = sr.ReadToEnd();
                    sr.Close();
                }
                return serialisedString;
            }
        }
    }
}

Hopefully others will find this useful.

 

Updated - A little more generic now – can serialize to either Json or Xml as needed by altering the type param.