About the Author:

With a career that has spanned advertising, production, technical services, and project management, Michael is able to articulate the wide range skills and professions that make the Internet work. This eclectic understanding and his desire to shine the light on those hiding behind techno babble has brought success to a wide range of projects.

Twitter @bissell

Linked In


Past Postings:

Using Dissent To Enhance Your Social Influence Online

Industry Profile - Author

Industry Profiles Full Time Employees - Professional Writer

Some Thoughts On Freelancing

Building Your Online Brand

Marissa Mayer and the Change in Yahoo's Remote Workforce

LinkedIn for Professional Writers

Fake Republican Twitter Accounts

"Did you mean?" -- Google's chiding nanny of search results

Branded Technology

Sharingspree.com -- Stealing more than GroupOn's Idea

The Internet Isn't Entertaining Enough

It's not your bank... It's Apple's and Amazon's

Violated by Madison Avenue

Google+ Scares Me

"We need to..." Internet Marketing Myths

Facebook's deal with the Devil

My cool new phone is a little too cool.

You are never alone

Promotion vs. Distribution... You'd think they'd know that one...

Publishing Industry Watch

Content for Social Media

Social Media Slot Machine

Anonymous vs Me

News from the Twitter Follow Campaign Trail

The art of Indiscriminate Twitter Following

The Cloudy Meaning of The Cloud

The Demand For The Loss of Creativity

Alien Technology and Government Conspiracies

Time for a New Reality

The Death of Email

Protecting Free Speech... Anonymously (and geekily)

Amazon Shouldn't Have Shut Down WikiLeaks

The Superpowers of the Hive Mind

Time for New Ideas

Comcast, Netflix and the Mystery of the Modem

The Great Technical Disconnect

New for the Sake of New

A Retail Store Built Like the Web

Disposable Personas

When did Google Start Policing the Internet?

Getting back to HTML basics, thanks to Apple

Inspecting my Navel Base

A shoebox vs. an online backup

Is Your "Resume" Website Recruiter-friendly?

iBooks -- Creative Epicenter or Gatekeeper?

The Failure of Success

The Economy is Going to Get Worse, but that's okay

Time lost on Twitter

Client Vendor Relationships

Twitter's back alleys and dark places

Social Media is NOT Advertising

Microsoft Courier

Form (designers) versus Function (geeks)

PDXBOOM -- The power of social media and the portland pipe bomb

China and Apple -- Different organizations, same management

The volume of screens

Logorama

Google Adds Biking Directions to Maps

Transmedia

That magical little tablet

How your website can be in two places at once

Masterpieces created by sheer volume

Suing over lack of originality

A Primer on Internet Fame -- dancing babies, hamsters, numa numa, and more...

Checking my messages

Rules are made to be broken -- in a reasoned, systematic way

So many accounts, so few passwords

Who really uses Twitter? 60% of Twitter's traffic isn't on Twitter

The Web is a Jerry Rigged Kludge

Twitter: Asleep at the Mouse Wheel

Where regulation is good: Google Voice and Vonage

How Facebook is (unintentionally) forcing programmers to piss off users

The Twit Cleaner

Perfect Secretary's pitch for @Adbroad (and the Youtube API)

The Emotions of Text

The Shorty Awards Scandal -- Manual Spam is still Spam

Google Analytics, the cloud and missing numbers #fail

Helen Klein Ross & Michael Bissell Interview at Adweek's Social Media Strategies Conference

The Internet is the New 60's

Cougars from New Zealand (and I don't mean big cats)

Adding facts together, or why you can't charge your cell phone from wifi

Social Media and the Destruction of the World

Rabid Fans vs Passive Viewers -- The Coco vs Leno saga

How to tell someone to retweet (without using up your 140 characters)

You can't buy social media

A book unopened is but a block of paper

Building the LOST: The Final Season Sweepstakes

Holiday SPAM (or the lack thereof)

Archiving Twitter

Too Many Toolbars

Random Censorship with Google Adwords

Accessibility and Shopping Online

Twisted path to customer service

Flash: Shiny objects blinding your audience

Twollow and other gold rush scripts

GPS in a Laptop computer

Thinking outside the box... There was a box?

Twitter was designed for Text Messaging

It's not the corporations, damnit

Entrepreneur or Dreamer?

Adweek Social Media Twitter for Brands Presentation

Socializing is more than Social Media

Generational Marketing is a Myth (or Who's your Daddy?)

Social Media is Just the Way We Use the Internet

Twitter Followers Don't Matter (ask the porn sites)

The Internet is Gooder than Books

Sometimes you don't want your campaign to go viral

Best Twitter Branding Campaign

Like flies to crap, Spammy Twitter Followers don't really go away

iPhone SMS Security Hole

How Flipmytweet works

Cell Phones as Microscopes

Digg is not the Hijacker -- You Are

Steve Ballmer -- the walking dead?

Twitter as an open mic poetry reading

Automatic Social [un]Awareness

First splash for United Against Malaria

New Media/Old Media and the CLIO Awards

Interview at SXSW: Mad Men Twitter And Tracking

We've got an App for that -- it's called the Web

Understanding Google To Get Your Resume Noticed

The trouble with Wordpress and other templates

Wayward Words with Baggage

Speaking at SXSW March 17th

The fleeting Memory of the Internet

It's okay to say 'I don't know'

Nike Takes Over Conquent

Facebook owns this title

Excuses, excuses

A little on Social Media

Feeding on Content

Attack of the Bots

Web 1.0

Net Neutrality

Getting clever with data feeds

The Other Credit Crisis

The Broadband Inauguration

T-Mobile owns Magenta and Other Patent Stories

The Risk-takers, Doers and Makers of Things

The noise of 20,000+ Twitter Followers

30,000 feet, 500 MPH Suburban Strip Mall

Cellphones, toilets and the Inauguration

The End of Days (of song): Microsoft Songsmith Example

Browser Bigotry

The Death of your Soul: Microsoft Songsmith

Creative Development or Developing Creatively?

The Myth of Wikipedia (or the Wiki-1400)

Online/Offline Sales -- is it really that bad?

Is PayPal Tacky?

Old School Web Design Still Works

Domain Squatting

Green Chri$tma$

QA 101

Portland Snow

Get some return on that web traffic

I think they have a backup...

I'd love to have that problem

The [un]importance of statistics

Don't be a tool of viral marketing

Emails, discussions, blogs, wiki and web content

You Designed for Print First

You let someone else register your domain name

You figured .biz, .info, .us would work fine

What's after the Integrated Circuit?

Intelligent life is out there (but it's bugger all down here on earth)

Subject Matter Experts Talking Other Subject Matter

The Totalitarian Regime of Apple

Oversimplifying how people work

crowdSPRING

Creative Services for the New World

Reverse Anthropomorphism

The End of Time

Better Living Through Twitter

Lessons Learned From Apple

It's the Brand, Baby

Business Architecture vs. Web Construction

On Truth

Inverse Peter Principle

Random Knowledge

The Hive



RSS for this blog
Getting clever with data feeds
Posted: 2009-01-26 13:07:49
Shortcut URL: http://t.conquent.com/y200



There is a lot of data available out there on the Internet these days, and more and more of this data is being made available in various forms of feeds or APIs (Application Programming Interface). For kicks, we wrote a feed from Twitter to Conquent so you can see live results scroll by at http://conquent.com/scroller/ and I also display my recent blog postings and Twitter postings through RSS on various portal pages.

But let's talk about the value for business.

Our accessible shopping comparison engine at http://empower.conquent.com/ pulls the literally millions of records from Shopzilla and other data sources and formats the results in a way that's easy for screen readers like JAWS or Window-Eyes to read as well as making it easier for mobility impaired, or vision impaired visitors without readers, to view the results.

The problem is with querying millions of items "on the fly." You can't do it efficiently -- even with solid bandwidth and good servers, pulling down millions of records takes time to process. And hitting the source data every time someone wants to see vacuum cleaners makes the user experience crawl and eventually lay down and die.

We worked around this problem with an "on-demand caching" of results. Shopzilla updates their system every 24 hours, so there's no reason to grab a set of data if we've seen it in the last 24 hours. The first time someone hits a category that hasn't been viewed in 24 hours, we fire up the process to grab the data, process it and cache it -- it's a little slower for that one person, but for the rest of the day it's lightning fast.

At the same time, we don't have to store unused categories -- if no one ever browses "tires" on empower.conquent.com, we haven't ever wasted bandwidth and processor time by grabbing something no one cares about. Granted, every time Google or MSN comes by they'll trip the category, but that works to our advantage creating a slow, background cache copy in the event someone DOES come by -- but if we don't already have it, the visitor will always get the most recent copy, regardless of whether we already grabbed it or not.

Building in these simple efficiencies makes the site a lot more scalable, faster, and keeps our content provider happier as we're not crowding them with unnecessary requests.

Next
Net Neutrality
Previous
The Other Credit Crisis


Comment on this blog
Your name:


Your email (will not be displayed):


Subject


Message



Enter the text above to help us filter spam: