-- UPDATED  Jan. 7 2004 --

If you don't have a clue where to start looking for the broken object(s), you need 
a solution that searches for them. Below I discuss two approaches, fsrefs.py and a 
recursive script based on the above recipe.

1) fsrefs.py should find ALL POSKeyErrors in a FileStorage, as Dieter Maurer 
reminded me on zodb-dev. But it only tells you the OID so you have to do a 
bit more work to find the object in the zope heirarchy.

If fsrefs.py doesn't come with your version of zope, you can find it 
by downloading ZODB3 (for zope 2.6, get ZODB3-3.1.2 or later), and look 
in the Tools directory. To run it you have to set PYTHONPATH to include 
$ZOPE/lib/python.

fsrefs.py will tell you the OID of any object that can't be loaded. In 
some cases, these seem not to be referred to by anything and it seems that
 these can be fixed by a pack. In other cases, one object refers to 
another object that is missing. This causes the first object to be broken.
In this case, packing may not help and you may need to manually delete the 
referring object.

Now that we have an OID, let's resolve that to the actual object like so:

>>> app._p_jar[oid] 

Note that fsfrefs.py prints OIDs like '0x1e9af', but 
app._p_jar[oid] needs the oid like '\x00\x00\x00\x00\x00\x01\xe9\xaf'.
If you don't feel like working out the conversion by hand,
try importing p64 from ZODB.utils.
It will convert an integer into a correctly packed string.

Once you have the object, you can start getting clues about it. It's not 
acquisition-wrapped so you can't easily find out its containment path, 
but you can find out its ID, mod time, type, and owner, like so:

>>> obj = app._p_jar[oid]
>>> print obj.getId()
>>> print obj.bobobase_modification_time()
>>> print obj.meta_type
>>> print obj.__ac_local_roles__

Armed with that information, you can use the Find tab in zope and 
should be able to easily track down the object. Once you know the path, 
you can delete it, and a subsequent pack should remove all traces of the 
offending object. Yay!


2) Recursive script approach:

Below is an alternative to fsrefs.py which I cooked up when I forgot 
about fsrefs.py. It has the advantage of printing the physical path to
broken objects that it finds, so you don't have to do any further detective 
work; but unlike fsrefs.py, this script only recurses ObjectManagers and 
might miss other kinds of references (notably image & file Data references),
so fsrefs.py might find problems that this function misses.
 
Define the following function, import and run it in a zope debugging session. 
This function is likely to take a looong time on a big database.
It will search the whole tree, unless you set stop_on_first true in which case 
it exits on the first exception encountered.

(fixes from Russ Ferriday added 10/22/03)

from ZODB.POSException import POSKeyError
def error_finder(folder, exception=POSKeyError, stop_on_first=None):
    """ start at the given folderish object.
    If stop_on_first is true, quit after one exception;
    otherwise, keep going through the whole tree."""
    for id, next_item in folder.objectItems():
        try:
            next_item.getId()
        except exception:
            print `exception`, "in folder",
            print  '/'.join(folder.getPhysicalpath()),     
            print "at id:", id
            if stop_on_first:
                raise "done"   # hack to break out of recursion
        else:
            # no error, recurse if it's objectManagerish
            if hasattr(next_item.aq_base, 'objectItems'):
                error_finder(next_item, exception, stop_on_first)

2.7+ will require a valid configuration setup:: 
 
 >>> from Zope.Startup.run import configure; 
configure('etc/zope.conf') 
 >>> from Zope import app; root = app() 
 >>> obj= 
root.unrestrictedTraverse('CGPublisher/publishers/13/proposals') 
 
and then the rest as given in the entry above.