Weaving a tangled web

Aug 3, 2009  |  Michael Wurzer

Brian Larson posted a detailed conclusion to his IDX series the other day, concluding that he’d recommend the NAR modify the IDX policy to:

  • Define what we mean by ‘web search engine,’ identify the benign uses they make of listing data, and incorporate those descriptions into the rules.
  • Say that broker IDX sites may allow and even encourage indexing by web search engines.
  • If brokers are particularly miffed by the “Nancy Smith” example, MLSs can prohibit display of listing agent in IDX (as long as state law does not require it).
  • Educate all brokers about how site indexing works and about technology options to allow them to take advantage of it.
  • Allow MLSs to adopt a rule requiring IDX sites to display terms of use prominently on the site (but giving brokers a few months’ grace period to implement them).
  • Develop a good model TOU and invite the MLSs to promulgate it to brokers; two key terms would make MLS a third-party beneficiary and would allow ‘web search engine’ use but not any other commercial use of the listing data.

I basically agree with all of this, though I think defining “web search engine” and benign uses is very difficult if not impossible to do when it comes to the web and why I think the terms of use Brian mentions are most important.

Yesterday, Rob Hahn followed up on Brian’s post with “Great Expectations, or On the Purpose of IDX“, in which he makes at least two key points:

  • Purpose of IDX.  Rob agrees with Brian that defining the purpose of IDX is critical: “Without this purpose statement, one could reasonably claim that the purpose of IDX is to benefit sellers as much as possible, or that IDX is meant to empower agents to be more competitive, or whatever.”
  • The Game is Over for IDX.  Rob makes the case that big brokers no longer benefit from IDX in the world of Google dominance, and speculates that the next model might be for big brokers to pull out of IDX, build great VOWs for themselves, and then encourage their MLS to build a great public web site to drive traffic to their VOWs.

I have a couple of comments in response to these ideas from Rob and the IDX issue in general:

  • VOWs.  As I posted a few weeks ago, establishment of VOWs as required by the DOJ/NAR settlement raise questions about MLS IDX policies.  On the one hand, many brokers and MLSs filled the vacuum created by the NAR/DOJ litigation by expanding IDX to include lots of data, including sold information.  In other words, IDX data feeds in these MLSs look a lot like VOW feeds.  On the other hand, some MLSs, like MIBOR, continue to have a restricted view of IDX and want to keep it off the web at least in terms of indexing by search engines.  The distinction between these views seems to be squarely in front of the NAR MLS Policy Committee for investigation and possible decision this fall as a result of the IDX/scraping issue raised by MIBOR this last spring.
  • Purpose of IDX.  I’ve stated before that investment in IDX by many industry participants makes changing that policy of concern, and that the NAR should not combine the VOW and IDX policies.  This brings us back to the purpose of IDX.  Though I agree with both Rob and Brian that a stated purpose would be useful in interpreting the policy and for brokers in deciding whether to opt in or out, I also think that any “purpose” of IDX only matters to the extent it is codified in the terms of use.  After all, the end result of IDX is to put listings on the web.  The web is an evolving, changing flow of information and any purpose stated today likely will change tomorrow.  As Rob points out, Google now dominates and defines the web in, perhaps, too many ways, but tomorrow it may be different.  That’s the beauty and creativity of the web and all its participants.  Declaring a purpose for IDX is useful, but, in the end, the result is that IDX allows agents to put others’ listings on their world wide web site, which, by definition, is open to the world.  Controlling that is going to be difficult, at best, and quite probably impossible.
  • Re-thinking IDX.  The open nature of the web (IDX) and the closed nature of VOWs raises the question as to whether the current IDX policies of MLSs recognize these differences.  I’m hopeful that Rob’s idea about IDX going away in favor of VOWs is a stretch too far, because, as mentioned above, IDX is a vibrant part of so many franchise, broker and agent strategies today.  At the same time, the industry needs to recognize the reality of VOWs and how they interplay with IDX.
  • Syndication.  One example of the evolution of the web is syndication.  Today, many franchises, brokers, agents and MLSs are syndicating listings to sites like Zillow, Trulia, Yahoo!, Google and many others.  One of the more interesting twists on this phenomenon is RealBird’s “IDX” solution based on listings in Google Base.  Tying these altogether, let’s say that Rob’s vision of big brokers pulling out of IDX in favor of VOWs coupled with an MLS consumer portal comes true.  Unless some terms of use somehow prohibit it, RealBird will continue to offer agent and broker “IDX” solutions based on the listings many MLSs are syndicating to Google Base.  This is just one of many reasons (the possibilities of a platform like Google Wave are another) why crafting a terms of use for the open web is so important and difficult, and why the industry, intended or not, is weaving a tangled web.
  • Standards. (You didn’t think I’d get through a post without mentioning standards, did you?)  One possibility for tying some of these issues together is to focus on the syndication standard developed over the last year or so.  At a basic level, syndication is very similar to IDX in that both are focused on advertising listings.  If that’s true, wouldn’t focusing IDX policy on the same data fields identified in the syndication standard make sense?  If we could standardize IDX on the syndication data set, that also would solve many of the cost issues associated with processing so many disparate IDX feed formats.  Having a standard format for IDX also would make crafting a more standard terms of use easier.  Getting there will be challenging, of course, particularly given the wide adoption of IDX, but the long-term payoff could be significant for the entire industry.

I’m heading off to San Francisco tomorrow for the Inman Connect conference, where the issue of VOWs will be front and center for the panel called “From Listing Data to People Data: The Next Challenge in VOWs and MLS Consumer Sites.” Join me there for the continuing discussion or leave a comment below.

17 Responses to “Weaving a tangled web”

  1. Rob Hahn says:

    Michael –

    Sage, detailed, and awesome as usual. I’ll have to reflect on what you said, and looking forward to speaking with you in person later this week.

    One thing, however, that strikes me as worthy of questioning is the value of Syndication in the Age of Google. Whether a broker syndicates listings to Trulia, Zillow, whoever… if the starting point for all search is still Google (and I submit that it is), then what’s the point of syndication at all? Google –> Trulia –> Broker Website vs. Google –> Broker Website or Google –> MLS –> Broker Website all result in about the same thing.

    But the latter two prevents various schemes of IDX-fishing and the like.

    What am I missing?


  2. I suspect Spencer or Sami or someone else will be able to make a case for why Zillow and Trulia are relevant in the age of Google. 🙂

    My overall point is that there’s a big distinction to be made between listings out on the open web (whether it’s through IDX or syndication or some other method) and listings in a VOW (the closed web). I think the industry needs to figure out a way to be comfortable with having listing advertisements (IDX/syndication) on the open web in all its glory (be it an MLS site, Google, Zillow, Trulia or others), and let innovation develop openly on that listing advertisement data set. You and I can prognosticate about which company will come up with the best way to drive traffic and sell houses from that data, but the market and the web will figure it out for sure if it is allowed to do so.

    At the same time, VOWs offer a great opportunity for franchises, brokers, agents and MLSs to improve service to consumers by providing a platform for communication and engagement when the consumer is ready. Importantly, even though the term closed web carries a potentially negative connotation, I don’t have that same view. Facebook is the most prominent example of the closed web, and it’s wildly successful because it provides a platform for communication on which millions are comfortable. Providing such a platform has been central to the MLS for a long time and that’s why I think VOWs are critical to the MLS of the future.

    That VOWs are important, however, doesn’t negate IDX or, more generally, publishing listings on the open web. That’s going to continue to happen and, in my view, focusing on one company such as Google or any other (whether they are dominant now or not) from a policy perspective is not the right approach.

  3. Matt Lavallee says:

    I’m going to take an antagonistic approach and say that perhaps IDX has flown too far from its intended purpose (which I’m basically improvising for lack of a published intent).

    Surmise with me that the purpose of “clearly visible” broker reciprocity was to level the reuse of the listing as a free advertisement. The practically-pre-web result was that the consumer had obvious steps to access the owner of the listing.

    Nowadays, what with the re-re-rescraping and micro-fine-print acknowledgement of the owning broker, things have gone a bit astray.

    So, based on my interpretation of the premise and the eventual misuse of the construct, I’m going to make a fresh recommendation along those lines:
    IDX listings must provide a trackback link to the owning brokerage (as provided in the IDX feed) with the owning brokerage’s name as the link text and said link in the same typeface, size, and color as the listing’s address.

    To me, this would restore the intent of IDX as cooperative advertising and dissuade the unsavory efforts of some to use a broker’s own listings as a (sick) tool for reselling leads.


  4. Including a trackback link also would follow the syndication standard, I believe. Interesting idea. I wonder how search engines would treat such links for the host and destination? Link love is a good thing, generally.

  5. John Mijac says:

    I feel essential parts of this conversation are missing: the Sellers and Buyers. So much attention is focused on the ownership of a listing and the value its data has for the listing brokerage that the intent of the listing a property in the first place gets lost: to sell it.

    Real Estate is evidently still undergoing a sea change from that old Caveat Emptor, Seller-only representation; else the truth of this matter would be more evident. In today’s market, Buyers are driving sales and most are knowledgeable enough to insist on their own representation. Go ahead and provide traceback links to the owning brokerage; an increasingly large crowd will insist on doing things their way anyway. In truth, it may not be long before the days of dual agency come to an (enforced) end. In such a world, the value of an IDX listing will be what it should be: to advertise a listing to potential Buyers and Buyer Brokers until it sells. I argue this outcome would be better for all. Such a world would include VOWs, primarily on the Buyer Broker side, IDX sites on the Listing side and the mls in the middle.

    Let Google try to compile all the data in the world: it cannot. There will still be pools of information which cannot be indexed, data which should not be indexed for general dissemination. Intranets like Facebook, like many commercial listing services . . . like the current MLS. VOWs are a good idea which will offer a better paradigm for Buyers, Sellers and Brokers on both sides.

  6. Matt Lavallee says:

    With all due respect, John, I disagree with the argument that the advocacy of buyer agency is the ideal winning scenario: The only result is a flip-flop of who controls the market (he with the content or he with the eyeballs). So far, the free market concept relies on the seller of goods having discretion over price, reach, message, and the investment value of advertising and partnerships.

    Flooding the public with “all the data, everywhere, all the time” would be a boon to buyer agency, but would also be a catastrophe for the market. I shudder to think of the endless misuse of content that would cause… especially the reselling of leads (which I think is flat-out dirty pool).

    I see the whole industry as more of a growing supply chain problem. The manufacturer wants to get broad reach, but also wants to know that its product is being properly represented, fairly positioned, and reaching its target consumers without wasting time, effort, or money. Some leverage resellers who target a market, some sell directly as wholesalers, and many do both and accept structured returns based on the eventual selling channel. Rarely (if ever?) are there multiple durable goods outlets that offer identical products and survive. Car dealerships are a decent example of that system just not working.

    The originator of the content (the seller & seller’s agent) — as in copyright law — have every right to decide where the listing goes and how it is used.


  7. I continue to take the middle ground: IDX and VOWs will co-exist and support each other. They are distinct and inter-related. They both support both buyers and sellers and their respective agents. The policy around VOWs is now well-settled, if not well-implemented. The policy around IDX is well-implemented, if not well-settled. The question I have is whether the two policies could or should be read or re-read in light of each other.

    Frankly, a perfectly logical outcome is the one pursued by several MLSs already of making IDX feeds as robust as VOW feeds. This was the reaction most had when the VOW settlement occurred — yawn, VOWs are dead, long live IDX and syndication. Importantly, however, many brokers and agents deploying IDX sites will create VOW-like registration requirements and the market will work out what model survives over the long term. Relying on the market makes sense to me and IDX and syndication certainly have the momentum right now.

    My purpose in writing this post, however, is to point out that VOWs could very well provide a way for brokers and MLSs to re-consider IDX in new ways that answer some of the questions posed by search engine indexing and other open web challenges. The terms of use for IDX data have not yet been written, but hte issue is squarely confronting the industry.

    Regardless of whether IDX feeds become more like VOWs or if VOWs become more important, I think it’s critical that listing data continue to be available in some form on the open web to foster innovation.

  8. @Michael/Rob:
    Darn! So much good stuff in both of your posts, but I need a few days to think these ideas over! (Anyone got a spare few days?)

    Your ideas touch on some broader strategic issues for MLSs and the brokers they are in business to serve…


  9. @Mike:
    You wrote: “I basically agree with all of this, though I think defining “web search engine” and benign uses is very difficult if not impossible to do when it comes to the web and why I think the terms of use Brian mentions are most important.”

    I can read this at least two ways:
    1.) The terms of use are “important” because they will include the definition of “web search engine” and benign (permitted uses). OR
    2.) The terms of use are “important” because they are an ALERNATIVE to defining “web search engine” and benign uses.

    I agree with you wholeheartedly, if you sense is (1). I cannot disagree more if your sense is (2). For a terms of use document to be effective at all, it will have to define the scope of permitted uses. And if “web search engine indexing” will be permitted uses, then we’ll have to define it IN the terms of use (as well as in the rules).


  10. Rich Bailey says:

    Michael, Rob, Brian, and the rest – Love this topic and really enjoy the dialogue. It’s ultimately healthy to raise and discuss these issues, so keep writing and I’ll keep reading (and referring people to these blogs).

    One more INMAN session worth checking out, since Michael you did raise the issue of standardization, is “MLS Hell: Coping with Data Normalization in an Abnormal Industry”, which is tomorrow at 10am Pacific.

    I hope to see all 3 of you there.


  11. Rob Hahn says:

    This is a fantastic conversation indeed; I’m looking forward to learning more from Mike, Matt, Brian and the others. And I’ll see you at Inman, Rich.


  12. John Mijac says:

    When we syndicate data we have an agreement with every syndicator that receives a feed. This agreement covers the terms of use for our data and what is permitted. When Rob mentioned he couldn’t see a reason to syndicate beyond Google, our rationale is that WE should be the source for our data to all subsequent permitted sites, not a third party. We would have no hope of control whatsoever with Trulia or anyone else, if we had a single agreement with Google. Also, who knows, Google could be gone in a year at the rate things change.

    It is true that the genie is currently out of the bottle, but if better, more timely and cleaner data can be had through Agreements with and direct feeds from the mls we believe that eventually that will win the day.

  13. Matt Lavallee says:

    John — I couldn’t agree more; the notion of “listings as an advertisement” is diluted rather quickly when the third or fourth re-use outlets are using the data for nothing more than SEO and not crediting the source.


  14. Brian, I would draft the terms of use focusing more on how the data needs to be displayed and what can’t be done than what can be done. I really do think defining a “web search engine” is quite difficult, especially moving forward. The recent changed by Google on its real estate search is a good example. The lines seem to get blurry very fast and the purpose of the terms of use is to keep people from mis-using the data. Google’s maps TOU are a pretty good example I think, though, of course, they would need tailoring for real estate. http://maps.google.com/help/terms_maps.html

    I especially like this provision in the Universal Terms of Service: “4.3 As part of this continuing innovation, you acknowledge and agree that Google may stop (permanently or temporarily) providing the Services (or any features within the Services) to you or to users generally at Google’s sole discretion, without prior notice to you.”

    There are provisions in the terms of service that get somewhat specific, but basically Google is saying, go ahead, use our stuff, but don’t do anything wrong and we reserve sole discretion to define wrong and stop providing you the service at any time.

  15. @Mike: You paraphrased that Google TOU as saying “go ahead, use our stuff, but don’t do anything wrong and we reserve sole discretion to define wrong and stop providing you the service at any time.”

    That just does not work in the MLS environment… It’s basically a “we-know-it-when-we-see-it” approach… On any continuum of conduct between “obviously OK” and “obviously bad”, ten different MLSs are likely to choose ten different points as the boundary where conducts goes from OK to bad. And some MLSs will actually consider a certain type of conduct OK when committeed by one actor but bad when committed by another. Such decisions are likely to be made in an arbitrary fashion (at least in MLSs that are not our clients), and MLSs can ill-afford claims they are disfavoring one business model over another…


  16. Good points, Brian. My point really is just what you’ve said: brokers and MLSs likely won’t know it until they see it and so the most important provision for me in the TOU is going to be this catch all, which will be included no matter what specifics are provided.

  17. Greg Swann says:

    Wow. What a tangled web. The solution to every bit of this is to get rid of the co-broke. The idea of the MLS as a walled fortress on information is already dead. No amount of whistling past the graveyard is going to reverse the tide of events — information will be free. I hope FBS is devoting attention to delivering a free-market MLS solution — sooner rather than later. The stupidities that emerge from the co-broke are too pronounced to persist for much longer.